A stereo display system using shape from shading algorithm is an image conversion device connected between a monoscopic endoscope and a 3D monitor. The system applies the algorithm which generates a depth map for a 2D image of video frames. The algorithm first calculates a direction of a light source for the 2D image and based upon the information of light distribution and shading for the 2D image to generate the depth map. The depth map is used to calculate another view of the original 2D image by depth image based rendering algorithm in generation of stereoscopic images. After the new view is rendered, the stereo display system also needs to convert the display format of the stereoscopic images for different kinds of 3D displays. Base on this method, it is necessary to replace the whole monoscopic endoscope with a stereo-endoscope system and no modification is required for the monoscopic endoscope.