Dynamic recrop of images

So this e-commerce site I'm doing has thousands of images. Some unfortunately were non-square and then placed centered on a white background, even if the predominant colors were darker. Going through all of these would be a nightmare production task.

I wondered if I could programmatically detect these rectangular white regions, and replace them with a better chosen color. The problem is that some of the images are really on a white background, centered and shouldn't be touched.

I've written some code that comes in diagonally until it detects a pixel that isn't white is the next pixel. I then check a vertical and horizontal line from that point. If one direction is all white, I assume that it is a rectangular region. Same from the bottom right corner coming up.

If one of these checks fail in both directions I bail. If I detect different orientations on the top and the bottom I bail. But if I detect horizontal or vertical stripes of white I then step in one pixel from all four "corners" and then average the color values. I then use this color to fill the white regions.

I know that certain shapes (peanut like) can screw up this calculation. Still the color chosen for the fill in that circumstance will be decent (and only fill white regions). Also some of the images have dithered edges, possibly throwing off the calculation. Still this seems to work pretty well.

Click here to see what I'm talking about. The first three images are a diagram of how the code looks at various images. The next items are the full size image, image resized (with border) without this dynamic cropping applied, and image resized with dynamic cropping applied.