An external charging system for an Implantable Medical Device (IMD) is disclosed having a thermal diffuser proximate to the primary charging coil for distributing heat from the primary charging coil. In an example, the primary charging coil is mounted to a first side of a circuit board, and the thermal diffuser is also connected to the first side and in contact with the primary charging coil. In one example, the thermal diffuser is a plastic material, such as an acrylic pad, with a high thermal conductivity and a low electrical conductivity. The thermal diffuser may also contact temperature sensors mounted to the first side of the circuit board.