PROBLEM TO BE SOLVED: To obtain a decoloring agent composition having reduced irritating odor and capable of imparting sufficient brightness to the hair, and further to obtain a hair dye composition. SOLUTION: The decoloring agent composition usable by being mixed with an oxidizing agent contains (a) ammonia in an amount of 1.0-3.5 wt.% expressed in terms of 28 wt.% ammonia water, (b) 2.0-6.0 wt.% monoethanolamine, (c) 1.5-10.0 wt.% single nonionic surfactant, or surfactants consisting of 1.5-10.0 wt.% nonionic surfactant and <=1.5 wt.% cationic surfactant, (d) a higher alcohol, (e) an oily component, and (f) water. The hair dye composition contains (g) a dye besides the components. The decoloring composition and the hair dye composition are constituted so as to be each an oil-in-water type emulsion, and so as to contain each a crystal structure.