Disclosed is a plug-in accessory for operating a mobile device as an external controller for an Implantable Medical Device (IMD). The accessory includes a connector insertable into a port on the mobile device. Accessory circuitry can be powered by a battery or by the mobile device. An application on the mobile device in conjunction with the accessory configures the mobile phone for immediate use as an IMD external controller. When the accessory is inserted into the port or a switch on the accessory pressed, the application operates to validate the accessory to unlock the phone to secure the mobile device and to render a graphical user interface on the mobile device for communicating with the IMD. The accessory can additionally include telemetry circuitry and an antenna for communicating with the IMD, rather than using short-range communication means provided in the mobile device itself.