In SPECT reconstruction, multi-modal reconstruction is combined with model-based multi-energy image formation. The scatter modeling of the model-based image formation uses resampling to facilitate convolution with the scatter kernels while maintaining resolution for the multi-energy projection. This combination of multi-modal and model-based multi-energy image formation simultaneously addresses the inaccuracy of the image formation process for complicated energy spectra and image blurring due to degradation of resolution. Varying the reconstruction by iteration may provide some of the benefits while reducing computational burden.