A focus control device includes an image acquisition section that acquires a captured image that has been captured by an imaging optical system, an evaluation value calculation section that calculates a focus evaluation value based on the captured image, the focus evaluation value being used to evaluate an in-focus state of an object in the captured image, a determination information calculation section that calculates determination information based on the focus evaluation value, a determination section that determines whether to start or stop an autofocus control process based on the determination information, and a focus control section that starts or stops the autofocus control process based on a determination result of the determination section.