In one aspect, a system for reducing the vibrations transmitted to sensors mounted on a tillage implement may include an implement frame and a vision-based sensor coupled to the implement frame. The vision-based sensor may be oriented relative to the implement frame such that a portion of the ground is positioned within a field of view of the vision-based sensor. As such, the vision-based sensor may be configured to capture image data of the soil positioned within the field of view. The system may also include a damper coupled between the vision-based sensor and the implement frame. The damper may be configured to allow relative movement between the vision-based sensor and the frame to reduce a magnitude of the vibrations transferred from the implement frame to the vision-based sensor.