A system (100) for monitoring an orientation of an agricultural implement (10) during an agricultural operation is disclosed. The system (100) includes a work vehicle (12) and an agricultural implement (10) coupled to the work vehicle (12) and configured to be towed by the work vehicle (12). The system (100) includes a vision-based sensor (108) coupled to the work vehicle (12) and a controller (102) communicatively coupled to the vision-based sensor (108). The controller (102) includes a processor (104) and associated memory (106). The memory (106) stores instructions that, when executed by the processor (104), configure the controller (102) to receive image data from the vision-based sensor (108); determine an orientation parameter based on the received image data; and initiate a corrective action based on the orientation parameter of the agricultural implement (10) while the agricultural implement (10) is being towed by the work vehicle (12). The orientation parameter describes an orientation of the agricultural implement (10) relative to the work vehicle (12).