A visual tracking and annotation system for surgical intervention includes an image acquisition and display system arranged to obtain image streams of a surgical region of interest and of a surgical instrument proximate the surgical region of interest and to display acquired images to a user； a tracking system configured to track the surgical instrument relative to the surgical region of interest； a data storage system in communication with the image acquisition and display system and the tracking system； and a data processing system in communication with the data storage system, the image acquisition and display system and the tracking system. The data processing system is configured to annotate images displayed to the user in response to an input signal from the user.
