Systems and methods for detecting an anomaly in an image from a set of images captured in vivo by an in-vivo imaging system may include, for each pixel of the image, associating the pixel with a color histogram value from a color histogram database； determining, for each pixel, whether the color histogram value associated with the pixel exceeds a histogram value threshold； assigning a pixel status to each pixel indicating whether the pixel is anomalous or normal； identifying one or more groups of adjacent anomalous pixels, the one or more groups of adjacent anomalous pixels each having a pixel size that exceeds a pixel size threshold； generating, using at least the one or more groups of adjacent anomalous pixels, a binary mask for the image； and determining an image anomaly score for the image based at least in part on the binary mask.
