A system for monitoring dietary activity of a user includes a wearable device having at least one audio input unit configured to record an audio sample corresponding to audio from a user's neck. The system further includes a processor configured to execute programmed instructions stored in a memory to obtain an audio sample from the audio input unit of a wearable device, determine segmental feature values of a set of selected features from the audio sample by extracting short-term features in the set of selected features from the audio sample and determining the segmental feature values of the set of selected features from the extracted short-term features. The processor is further configured to, using a classifier, classify a dietary activity based on the determined segmental feature values of the audio sample and generate an output corresponding to the classified dietary activity.