There are provided an endoscope system and a method of operating the endoscope system that can set the display contents of each observation image without applying a load to the entire device to emphasize only specific subject information in a case where a plurality of kinds of illumination light are emitted while being switched and observation images corresponding to the plurality of kinds of illumination light are displayed while being switched.A plurality of kinds of illumination light are emitted while being switched according to a specific light emission pattern. A plurality of observation images, which are obtained from image pickup of an object to be observed illuminated with each illumination light, are acquired. Control to display the plurality of observation images on a display unit while switching the plurality of observation images according to a specific display pattern is performed. The specific light emission pattern is fixed and the specific display pattern is changeable.