What is disclosed is a system and method for determining a respiration rate from a video of a subject breathing. One embodiment of the present method involves the following. A video is received of a subject breathing which comprises a first portion of N image frames, and a second portion of M image frames, N+M=T and N≧10 seconds of video. For each image frame of the first portion, flow vectors Ft are determined for each (x,y) pixel location. A correlated flow field V is then calculated for the first portion of video. For each image frame of the second portion, flow vectors Ft(x,y) are determined for each (x,y) pixel location and a projection of Ft along V is calculated to obtain a velocity of thoracoabdominal motion in the direction of V. The velocity is integrated to obtain an integrated signal. Respiration rate is determined from the integrated signal.