A deep machine-learning approach is used for medical image fusion by a medical imaging system. This one approach may be used for different applications. For a given application, the same deep learning is used but with different application-specific training data. The resulting deep-learnt classifier provides a reduced feature vector in response to input of intensities of one image and displacement vectors for patches of the one image relative to another image. The output feature vector is used to determine the deformation for medical image fusion.
