A method fits primitive shapes to a set of three-dimensional (3D) points by first converting the set of 3D points to a distance field. Each element in the distance field is associated with a distance to a nearest point in the set of 3D points. A set of two or more candidates are hypothesizing from the primitive shapes, and a score is determined for each candidate using the distance field. Then, the primitive shape to fit to the 3D points is selected from the candidates according to their scores.