An endoscope system which can reduce visual discomfort even in a case where a normal image and a special image are alternately displayed, and a method of operating the endoscope system are provided. In a case where an observation target including a first structure is illuminated with first mode illumination light, a first mode display image obtained by imaging the observation target is acquired. In a case where the observation target is illuminated with second mode illumination light, a second mode display image obtained by imaging the observation target is acquired. A corrected image is obtained by correcting the first mode display image or the second mode display image such that a difference in a color and/or brightness of the first structure between the first mode display image and the second mode display image before the correction becomes small.
