Described herein are systems and methods for detecting a physiological response based on thermal measurements while accounting for effects of the environment. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAMin) that takes thermal measurements of a region of interest (THROI) on a user's face, and an outward-facing head-mounted thermal camera (CAMout) that takes thermal measurements of the environment (THENV). CAMin does not occlude the region of interest, and the system further includes a computer that detects the physiological response based on THROI and THENV. Optionally, the computer generates feature values based on sets of THROI and THENV, and utilizes a machine learning-based model to detect, based on the feature values, the physiological response.