An endoscope system includes an image storage unit that stores three-dimensional image information of a subject acquired in advance, a luminal organ extraction unit that extracts a predetermined luminal organ from the three-dimensional image information, and an insertion unit of an endoscope The objective optical system provided at the distal end of the optical axis, the visual line direction detection unit for detecting the visual line direction by the objective optical system, and the core line direction and the visual line direction of the predetermined luminal organ extracted by the luminal organ extraction unit are compared. A direction comparison unit, and a position / orientation information recording unit that records information on the position and orientation of the distal end of the insertion unit of the endoscope when an angle formed between the core line direction and the viewing direction is equal to or less than a predetermined threshold.内視鏡システムは、予め取得した被検体における３次元画像情報を記憶する画像記憶部と、３次元画像情報から所定の管腔臓器を抽出する管腔臓器抽出部と、内視鏡の挿入部の先端に設けられた対物光学系と、対物光学系による視線方向を検出する視線方向検出部と、管腔臓器抽出部により抽出された所定の管腔臓器の芯線方向と視線方向とを比較する方向比較部と、芯線方向と視線方向とのなす角が所定の閾値以下の場合に内視鏡の挿入部の先端の位置及び姿勢の情報を記録する位置姿勢情報記録部と、を備える。
