Ultrasounds value in the neurosurgical operating room is maximized when fused with pre-operative images, The disclosed system enables real-time multimodal image fusion by estimating the ultrasounds pose with use of an image- based registration constrained by sensor measurements and pre-operative image data. Once the ultrasound data is collected and viewed, it can be used to update the pre-operative image, and make changes to the pre-operative plan. If a surgical navigation system is available for integration, the system has the capacity to produce a 3D ultrasound volume, probe-to-tracker calibration, as well as an optical-to-patient registration. This 3D ultrasound volume, and optical-to-patient registration can be updated with conventional deformable registration algorithms and tracked ultrasound data from the surgical navigation system. The system can also enable real-time image-guidance of tools visible under ultrasound by providing context from the registered pre-operative image when said tools are instrumented with sensors to help constrain their pose.
