A framework for image-based probe positioning is disclosed herein. The framework receives a current image from a probe. The current image is acquired by the probe within a structure of interest. The framework predicts a position of the probe and generates a recommendation of a next maneuver to be performed using the probe by applying the current image to a trained classifier. The framework then outputs the predicted position and the recommendation of the next maneuver.