A device for inserting a subcutaneous infusion device into skin of a patient. The device can include a housing (110), a needle hub (130) including a needle (336), a sleeve (140), and a spring (150) engaging the needle hub. The device can also include a cap (170) coupled to the housing, and a retention member (520, 540) configured to maintain the device in a ship state prior to decoupling of the cap from the housing. In one example, the retention member can include a boss (520) coupled to the cap and configured to engage the infusion device to maintain the infusion device in the ship state prior to decoupling of the cap from the housing. In another example, the retention member can include a tab (540) formed by the sleeve, the tab engaging a bead (510) on an internal surface of the cap when the sleeve moves relative to the housing while the device is in the ship state.