A system and method for efficiently and accurately classifying user activity. In a non-limiting example, accelerometer signals and/or gyroscope sensor signals may be analyzed to classify user activity, for example, that of a user of a handheld and/or wearable device. Information from additional sources and sensors (e.g., other inertial sensors, non-inertial sensors, passive sensors, etc.) may also be utilized.