A robotic system and methods are provided. The robotic system comprises a tool and a manipulator with links for moving the tool. A controller implements a virtual simulation wherein the tool is represented as a virtual volume interacting with a virtual boundary defined by a mesh of polygonal elements. The controller computes a reactive force responsive to penetration of polygonal elements by the virtual volume. The reactive force is computed based on a penetration factor being a function of a geometry of the virtual volume bound relative to a geometry of the polygonal element. The controller applies the reactive force to the virtual volume to reduce penetration of the polygonal element by the virtual volume. The controller commands the manipulator to move the tool in accordance with application of the reactive force to the virtual volume to constrain movement of the tool relative to the virtual boundary.
