Embodiments described herein provide various examples of a visual-haptic feedback system for generating a haptic feedback signal based on captured endoscopy images. In one aspect, the process for generating the haptic feedback signal includes the steps of: receiving an endoscopic video captured for a surgical procedure performed on a robotic surgical system； detecting a surgical task in the endoscopic video involving a given type of surgical tool-tissue interaction； selecting, a machine learning model constructed for analyzing the given type of surgical tool-tissue interaction； for a video image associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, applying the selected machine learning model to the video image to predict a strength level of the depicted surgical tool-tissue interaction； and then providing the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.
