A hand held robotic system that remains stiff so long as it is operating within allowed limits, but which become actively controlled once the operator exceeds those limits. The system thus corrects deviations by more than a predetermined amount of the operator's hand motions, so that the tool remains in the allowed region even when the operator's hand deviates from the planned trajectory. The pose and path of the robotic operating head is ascertained by means of a navigation or tracking system, or by means of a proximity device to measure the closeness of the operating head to a damage sensitive feature. As the tool deviates from its predetermined path or pose, or comes too close to the hazardous area, the robot control acts to move the tool back to its predetermined pose or path, or away from the hazardous region, independently of user's hand movement.
