Systems, methods, tangible non-transitory computer-readable media, and devices for operating an autonomous vehicle are provided. For example, a vehicle computing system can receive sensor data and vehicle data from sensors of a vehicle. The sensor data can be associated with states of passengers in a passenger compartment of the vehicle. The vehicle data can be associated with states of the vehicle. The vehicle computing system can determine, based on the sensor data and the vehicle data, when the states of the passengers are associated with health conditions. Actions for the vehicle to perform can be determined based on the health conditions associated with the passengers. Furthermore, the vehicle computing system can generate control signals to control performance of the actions by the vehicle.
