A portable temperature-altering device, which is fixed on fabrics worn by a user or the user himself, includes a main body, at least one temperature-altering element, and a combining unit. The main body is in an elongated-pouch shape and defines an accommodating space having at least one opening. The at least one temperature-altering element is disposed in the accommodating space of the main body for altering the temperature of the main body. The combining unit disposed on an external surface of the main body includes a first combining part adjacent to one end of the main body and a second combining part adjacent to the other end of the main body. Through mutually matching up the first combining part with the second combining part, the main body can be fixed on the fabrics worn by the user or on the user himself.