Bioinspired polarization navigation technology has become a new autonomous navigation solution in GNSS-denied environments. In the case of point-source polarization sensors, clouds will affect the accuracy of the sensors. To enhance the adaptability of the polarization navigation system in complex weather conditions, this article presents an integrated navigation system using the multidirectional polarization sensors, sun sensor, and inertial measurement unit (IMU). The information fusion framework is established for attitude and heading determination. The proposed outlier detection method in multidirectional polarization sensors is used to select the optimal polarization sensors. Moreover, a chi-square-based Kalman filter (Chi-KF) is adopted to adapt variable measurement noises of the polarization sensors and sun sensor. The results of static ground test and unmanned aerial vehicle (UAV) flight tests demonstrate that the proposed method can improve the navigation accuracy in cloudy weather conditions.