The subject disclosure is directed towards eye gaze detection based upon multiple cameras and/or light sources along with an adaptive homography mapping model. Learning of the model includes compensating for spatially-varying gaze errors and head pose dependent errors simultaneously in a unified framework. Aspects including training the model of adaptive homography offline using simulated data at various head positions.