An apparatus and method to recognize a user's emotion based on emotional segments are provided. An emotion recognition apparatus includes a sampling unit configured to extract sampling data from input data for emotion recognition. The emotion recognition apparatus further includes a data segment creator configured to segment the sampling data into a plurality of data segments. The emotion recognition apparatus further includes an emotional segment creator configured to create a plurality of emotional segments that include a plurality of emotions corresponding to each of the respective data segments.
