A personal audio device, such as a wireless telephone, generates an anti-noise signal from a microphone signal and injects the anti-noise signal into the speaker or other transducer output to cause cancellation of ambient audio sounds. The microphone measures the ambient environment, but also contains a component due to the transducer acoustic output. An adaptive filter is used to estimate the electro-acoustical path from the noise-canceling circuit through the transducer to the at least one microphone so that source audio can be removed from the microphone signal. A determination of the relative amount of the ambient sounds present in the microphone signal versus the amount of the transducer output of the source audio present in the microphone signal is made to determine whether to update the adaptive response.