For training for and performance of patient modeling from surface data in a medical system, a progressive multi-task model is used. Different tasks for scanning are provided, such as landmark estimation and patient pose estimation. One or more features learned for one task are used as fixed or constant features in the other task. This progressive approach based on shared features increases efficiency while avoiding reductions in accuracy for any given task.
