The present invention relates to a system (1, 1, 1") for determining a vital sign of a subject (100) by which the likelihood of generating and outputting false alarms is reduced. The system comprises a vital sign processor (30, 30a) for processing said vital sign information signal measured by a sensor (10, 10, 10a, 10b) attached to a subject to obtain a vital sign of said subject, an image analysis unit (50) for detecting motion of a marker (20, 20, 20a, 20b) attached to said sensor from image data obtained by an imaging unit (40) from at least an imaging region containing said sensor (10, 10, 10a, 10b), and an alarm unit (60) for generating and outputting an alarm signal based on the measured vital sign information signal and/or obtained vital sign and on the detected motion of said marker.