Provided are an image processing device, an endoscope system, an image processing method, and a program capable of automatically differentiating a medical image including a scene of interest and supporting saving of the medical image according to a differentiation result.The image processing device includes a medical image acquisition unit (41) that acquires a medical image, a scene-of-interest recognition unit (51) that recognizes a scene of interest from the medical image acquired using the medical image acquisition unit, a degree-of-similarity calculation unit (52) that, for the scene of interest recognized using the scene-of-interest recognition unit, calculates a degree of similarity between the medical image acquired using the medical image acquisition unit and a standard image determined for the scene of interest, and a saving processing unit (53) that executes processing for saving the medical image in a saving device based on the degree of similarity calculated using the degree-of-similarity calculation unit.
