A surgical robotic system automatically calibrates tubular and flexible surgical tools such as endoscopes. By accounting for nonlinear behavior of an endoscope, the surgical robotic system can accurately model motions of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. The surgical robotic system models the nonlinearities using sets of calibration parameters determined based on images captured by an image sensor of the endoscope. Calibration parameters can describe translational or rotational movements of the endoscope in one or more axis, e.g., pitch and yaw, as well as a slope, hysteresis, or dead zone value corresponding to the endoscope's motion. The endoscope can include tubular components referred to as a sheath and leader. An instrument device manipulator of the surgical robotic system actuates pull wires coupled to the sheath or the leader, which causes the endoscope to articulate.
