Medical endoscope system includes image pickup device and control target devices and controller configured to control the control target devices. The controller includes manipulator setting unit configured to set one candidate as manipulator from among candidates registered beforehand and to set the others as manipulation prohibited persons, object detection unit configured to detect object meeting prescribed condition in image from the pickup device, manipulator recognition unit configured to determine object that corresponds to a manipulator set by the manipulator setting unit from among the detected objects and to recognize the determined object as the manipulator, action detection unit configured to detect prescribed action of the determined object, control signal generation unit configured to generate control signal that controls a control target device in accordance with detection result by the action detection unit, and device control unit configured to control the control target device in accordance with the control signal.
