In one aspect, a system for controlling the operation of agricultural sprayers may a vision-based sensor configured to capture data indicative of one or more features within a field across which an agricultural sprayer is traveling. A controller of the system may be configured to identify the one or more features within the field based on the data received from the vision-based sensor and determine a position of a boom component of the agricultural sprayer relative to the one or more identified features. Furthermore, the controller may also be configured to initiate a control action to adjust the position of the boom component when it is determined that the relative position between the boom component and the one or more identified features is offset from a predetermined positional relationship defined for the boom component.
