An ultrasound (US) device (10) includes a US scanner (14) and a US probe (12) operatively connected to the US scanner. The US scanner and US probe are configured to acquire a succession of two-dimensional (2D) image frames of a portion of a patient. The device also includes a tracking sensor (28)； and at least one electronic processor (20) programmed to: acquire a reference three-dimensional (3D) image using the US scanner and US probe； track the pose of each 2D image frame relative to the pose of the last 2D image frame in the succession of 2D image frames based on tracking data acquired by the tracking sensor； and correct for drift of the trucked poses of the succession of 2D image frames by drift correction operations including: aligning a current 2D image frame with the reference 3D image to optimize a similarity metric measuring similarity of the current 2D image frame to an intersected slice of the reference 3D image； and updating the pose of the current 2D image frame to a corrected pose determined from the alignment of the current 2D image frame with the reference 3D image.
