Where's the table?

Background subtraction has gone fairly well. It seems that it will be a decent way to find the ball. Making a good background model will reduce noise in the background. Stereo vision will make it easier to differentiate the ball from the opponent. Looking at consecutive frames will narrow down the possible locations for the ball. So I'm moving on to the next question: where am I?

I realized that even if I could identify the ball, I needed to know where it was in relation to the table and in relation to the robot. I'm going to assume that the table is stationary, that the robot's base is stationary, and that the cameras are stationary. All plus or minus some noise when the wind blows and the camera shakes.

So that means that this problem only needs to be solved once when setting up, and that solution should remain valid. I suppose I could force my setup to be rigidly measured. E.g. the robot base is 20cm behind the table at the center line, 30cm above the table, always. Yeah, that could work. In fact, I will probably do it that way first. But it would be nice if my robot could be placed in front of the table and figure that out for himself. After all, ping pong tables are fairly distinctive objects. You'd think it would be easy to identify the table in the image, apply some stereo math, and come up with the 3D model of where the cameras and robot are with respect to the table.

New Image

Here is the input image I've decided to work with. I've blurred my handsome face because I'm a privacy freak and because I'm embarrassed that my forehead looks like a ping pong ball. It has some challenging conditions: patches of bright sunshine in the background that gets reflected in the surface of the table. It's kind of a pain in the ass. It makes me think that maybe robot researchers resort to carefully controlled lighting and matte surfaces on everything. But I'm better than that. Maybe.

As a human, it sure seems easy to identify the table. Not a lot of ambiguity, despite the challenging conditions. So shouldn't a machine be able to do this easily?

Detect The Color Blue

If I'm going to be using color images (still to be decided), I should make use of the color information. My table is blue, which makes it stand out. Annoyingly my shirt is a similar blue, and my jeans are not far off. How do you even identify "blue" from an image?

I went to Matlab to experiment. Since blue is one of the RGB colors, it should be easy. However, in RGB space, if you want B to be lighter or darker, you have to add some R and G, because there is no "light/dark" element to it. "Light" happens when you have more of every color, "dark" when you have less of every color. Here's what I came up with. In short, the B channel has to dominate the other colors.

function blue = blueDetect(img)
% split the colors for clarity
r = double(img(:,:,1));
g = double(img(:,:,2));
b = double(img(:,:,3));
% calculate blueness
blue = (b - r*0.5 - g*0.5) ./ b;
% display the image, normalizing so that the most blue pixel is all the way red
image( (blue ./ max(max(blue))) * 64);

And here's what it looks like:

This uses Matlab's default "jet" color scheme: the more "hot" the color, the more "blue" I've decided it is. (I'll use this color scheme in other images but I'll be excluding the scale from now on.) So it was kind of accurate. You can apply a threshold to give a yes/no answer as to whether each pixel was blue. This is using 30% "blueness" as the cutoff:

I think this shows the real problem: some of the table isn't blue. Go back and look at the original image. On the far side of the table, the reflection makes it more white/gray than blue. My blue detector doesn't even think it's close. That's going to be a problem. A human uses other clues to decide that it was a reflection, but apparently that's not easy to do. It also seemed to get confused by the back of the paddle on the right of the image. The paddle is black. To a human, it's pretty clearly black. But I guess the way I calculate things, it's more blue than red or green, so it is bucketed as being blue.

I decided to make another blue detector. This time I decided to use abandon the RGB color space, in favor of HSV. That stands for hue, saturation, value. "Value" seems to be approximately equivalent to "light" vs "dark". I'm going to give you three images, with each of HSV and broken down, so you can see how they behave.

So the good news is that the table is a very consistent hue value. The bad news is that the background wall is almost the same hue. As is the cement below the table. But when you look at the saturation, you can see that the table is a very saturated blue, but the wall and the cement are not. So it seems that it might be possible to combine this information to identify the blue of the table.

I chose to do this as a probability distribution. I create a normal distribution for each of H, S, V that says what the mean value of "table blue" is, and how far it is allowed to stray from that ideal. When evaluating a pixel, I multiply the probability from each of the distributions together to get a combined probability of being "table blue". Here's the code.

function blue = blueDetect2(img)
% convert from rgb to hsv color space
imghsv = rgb2hsv(img);

% define the means and standard deviations in each of H S V
% hue is very tight around 39/64ths
hueMean = 39.0 / 64.0;
hueStd = 1.8 / 64.0;
% saturation is high but wide to account for glare and shadow
satMean = 42.0 / 64.0;
satStd = 11.0 / 64.0;
% value is high and wide (but this captures most of the image)
valMean = 47.0 / 64.0;
valStd = 8.0 / 64.0;

% calculate the probability for each pixel for each component
hueFit = normpdf( (imghsv(:,:,1) - hueMean) ./ hueStd );
satFit = normpdf( (imghsv(:,:,2) - satMean) ./ satStd );
valFit = normpdf( (imghsv(:,:,3) - valMean) ./ valStd );
% combine the three component probabilities together
allFit = hueFit .* satFit .* valFit;

blue = allFit;
% display the image, normalizing so that the most blue pixel is all the way red
image( (blue ./ max(max(blue))) * 64);

And here's the resulting image:

That seems to be better than the previous method. It is more sure about the table, and doesn't seem distracted by the paddle. There is less "blueness" on the wall or the rocks in the bottom right. Here is a thresholded image at 30% blueness:

Conclusion

This entry explored ways to identify the table via color. It was moderately successful, but the challenging image meant that it wasn't perfect.

I still haven't identified the corners of the table, which would be necessary for my 3D model. And there are many other approaches I could try, aside from color, to identify the table. I leave that for other blog entries.