A system for providing automatic diagnosis and decision support includes: a medical image database； generative learning and modeling modules that build distributional appearance models and spatial relational models of organs or structures using images from the medical image database； a statistical whole-body atlas that includes one or more distributional appearance models and spatial relational models of organs or structure, in one or more whole-body imaging modalities, built by the generative learning and modeling modules； and discriminative learning and modeling modules that build two-class or multi-class classifiers for performing at least one of organ, structure or disease detection or segmentation.
