Images to be projected onto a projection surface are modified prior to projection based on color values sampled from the projection surface. A camera samples color values from the projection surface and a processor generates a modified source image based at least in part on pixel color values in an original source image and the color values sampled from the projection surface. A projector then projects the modified source image onto the projection surface.