Sunday, March 17, 2013

Small progress in triangulation

That last post gave me new emotional strength to approach the problem again. The effort actually paid off, with a partial solution to the problems introduced in my last post.

I can now rectify the images without them looking all weird. Here is the fixed version of the rectified images, side-by-side.

What was wrong? Well, like I suspected, it was a small error. Two small errors, actually, in how I was using the stereoRectify function. First, I was using the flag CV_CALIB_ZERO_DISPARITY. That's the default, so I figured it made sense. Nope. I cleared that flag and things got better. Second, I was specifying an alpha of 1.0. The intent of the alpha parameter is to decide how much black filler you see versus how many good pixels you crop. My answer of 1.0 was intended to keep all the good pixels and allow as much filler as necessary to get that done. I think that was causing the zoomed-out look of the rectification. I changed my answer there to -1 -- which is the default alpha -- and things got better. So I feel pretty good about grinding away until it worked.

I went a little further, and I also found out how to rectify points within the images. That has allowed me to map the table landmark points into the rectified images. You'd think that would be easy... and, in the end, it was. But I did it the hard way first. You see, the OpenCV functions to rectify the image (initUndistortRectifyMap and remap) actually work backwards: for each pixel in the rectified image, they calculate which pixel in the unrectified image to use. Whereas I now want to take specific pixels in the unrectified image, and find out what pixels those would be in the rectified image. That's opposite direction, and when your grasp on the math behind these functions is tenuous, it takes a while to reverse it. However, after solving it on my own, I discovered that the undistortPoints function has some optional arguments that also allow you to rectify the points at the same time. Anyway, those points are circles in these two rectified images:

Despite this progress, I still cannot triangulate. I assumed that fixing the rectification would also fix the triangulation, but this hasn't happened. In fact, my triangulation answers are unaltered by the fixes made in the rectification.

Even further, I also recreated the triangulation results using a different approach, to get the same (incorrect) answers. This time I used the disparity-to-depth "Q" matrix that stereoRectify produces, and feed it through perspectiveTransform. The answers are within a few mm of the triangulatePoints answers.

So, what's left to try? I have a suspicion that a mixture of left and right handed coordinates are to blame. So I'm going to try to push on that for a while, to see if it leads anywhere. My grasp of left and right handedness is flimsy and I have to keep referring to the wikipedia page.

After that, I'm buying at least one book on the math and logic that underlies all this 2D/3D vision stuff. I probably should have done that a month ago. I'm going to start with Hartley and Zisserman's "Multiple View Geometry in Computer Vision" which is apparently the bible of 3D vision, and I'll go from there.


  1. I did not have such problems with CV_CALIB_ZERO_DISPARITY. see for my results.

  2. Hi JB, thanks for these posts, very useful as i ran in the same problems. I find the whole rectification stuff in OpenCV extremely buggy and ill-documented. The whole problem is because you have your cameras at an angle. OpenCV wants them to look at a perpendicular direction, i.e., both to the front, and not angled at your table.
    If you do want to proceed like this, fiddle with the projection matrices, i.e., 2) -=, 2)*0.45; //Manually set cx, 2) -=, 2)*0.30; //Manually set cx

  3. Any updates on this?

    I have a similar problem with handedness.

    But no answers still...

  4. Hello JB,
    Hope you already have find a solution for your problem. Sorry i can not healp you, I wondered if YOU can healp me with something. Soo i also make a project that include two cameras and i also use two ps3 eye cameras for stereo vision. Can you please tell me how to connect ps3 cameras to the computer - i mean what drivers do i need and how to use it in open cv (more specifically emgu cv). Now i can not use both cameras at the same time only one and i can not figure how to use both at the same time. Can you please tell me how to proper connect and use it in code??


Be nice, remember I'm an amateur, but by all means please give me feedback!