Provided is a coloring agent composition to be added to a gypsum-containing raw material powder constituting a gypsum product and to be used for coloring the gypsum product, the coloring agent composition being a powder containing a pigment, calcined gypsum, and amorphous silica. A process for producing the coloring agent composition includes a first step of mixing a pigment, calcined gypsum, and amorphous silica to obtain a mixed powder. Use of the coloring agent composition makes it possible to easily produce gypsum products such as a powder and a hardened body that are excellent in colorability.