For material decomposition in medical imaging, a machine-learned model is trained to decompose. For example, spectral CT data for a plurality of locations is input, and the machine-learned model outputs the material composition. Using information from surrounding locations for the decomposition by the machine-learned model for a given location may allow for more accurate material decomposition and/or three or more material decomposition.