Realistic ultrasound imaging simulation requires modeling of scatterers corresponding to different imaging speckle appearances. A scatterer generator acquires a plurality of ultrasound signal samples, each corresponding to a different ultrasound capture and reconstructs a scatterer representation from the ultrasound signal samples and associated Point Spread Functions. Point Spread Functions (PSFs) may be estimated from multiple image acquisitions at the same reference position resulting from beam-steering. Reconstructed scatterers may then directly be used in ultrasound simulation or an additional step of modeling the scatterers may be applied. Statistical distribution parametrization or texture synthesis may be used to model the scatterers. Different scatterer models may be used for different homogeneous regions. The reconstructed scatterers and/or the scatterer models may be registered into a library of scatterers by the scatterer generator.