Lab: Epipolar Geometry

CSC 262 - Computer Vision - Weinman

Summary:

You will estimate the fundamental matrix and use it to plot the epipoles and epipolar lines in a stereo image pair.

Deliverables

• The Matlab script used to make your comparisons and generate your report
• The Matlab workspace file containing your correspondences (A.4)
• (5 points) Rank of the initial fundamental matrix estimate (C.4)
• (10 points) Singular values of the initial fundamental matrix estimate and commentary (C.6)
• (5 points) Adjusted fundamental matrix rank (C.8)
• (10 points) Image of left point and right epipolar line (D.1)
• (10 points) Image of right point and left epipolar line (D.2)
• (10 points) Observations on matching along epipolar line (D.3)
• (10 points) Image of all right epipolar lines (D.4)
• (10 points) Observations of epipolar geometry (D.5, D.6)
• (10 points) Professionalism of write-up

Extra Credit

• (10 points) Automated point correspondences

Preparation

Load the stereo image pair for this lab from the MathLAN. You need not convert to grayscale or double.

/home/weinman/courses/CSC262/images/left.png

/home/weinman/courses/CSC262/images/right.png

Exercises

A. Getting Correspondences

At the outset, we will get point correspondences manually. In the future you may want to see if your MOPS implementation can do it automatically.

1. Show the left and right images in two separate figures. Use figure(n) where n is the number 1 for the left image and 2 for the right image, so that you get predictable figure numbers.
2. According to the Matlab documentation, the command "ginput enables you to select points from the figure using the mouse for cursor positioning. The figure must have focus before ginput receives input." For example,

   [xx,yy] = ginput(1);

   places a cross-hairs on the current figure. Once you click, the coordinates of the click are returned, with the column in xx, and the row in yy. Write a for loop to gather at least eight points using the following steps:
   1. Display the left and right images in their appointed figure windows.
   2. Focus the figure containing the left image (use figure(n), where n is the figure number you wish to focus).
   3. Allow the user to select a point in the left image.
   4. Add the selected point to two vectors, one containing x-coordinates and the other containing y-coordinates of points from the left image.
   5. Plot the selected point on the left image using a plus in a color you find easy to see.
   6. Focus the figure containing the right image.
   7. Allow the user to select a point in the right image.
   8. Add the selected point to two similar vectors of corresponding points from the right image.
3. Use your loop to find at least eight point correspondences (choosing 20 will make your results more accurate; beware the reliability of conclusions drawn from inaccurate results due to too few matches).
   Important notes:
   1. Corners are easier to localize accurately.
   2. Try to distribute your points throughout the image.
   3. Try to distribute your points in scene depth.
   4. Make sure the points you select are not all coplanar.
   5. : Be as precise as possible in localizing your corners; many (even small) errors will yield a very poor fundamental matrix estimate (and a poorer grade).
4. Save the points you created in a Matlab workspace so that you can repeat other aspects of the lab without having to do this again. Use save.
5. I recommend visualizing your points now to verify that they are in fact all proper correspondences. (If they're not, you'll get very poor results later.)

B. Constructing the System Matrix

1. Load the correspondences you found in part A from your saved Matlab workspace file.
2. Construct the P×9 system matrix using the correspondences. Recall that each row has the form
   

   [ xLxR,  xLyR,  xL,  yLxR,  yLyR,  yL,  xR,  yR,  1 ]

C. Estimating the Fundamental Matrix

1. Apply singular value decomposition (SVD) to your system matrix.
2. Extract the column from V corresponding to the smallest singular value. (Matlab orders the singular values from largest to smallest for you).
3. Use reshape to make the 9×1 column into a 3×3 matrix, which is your initial estimate of the fundamental matrix F.
4. What is the rank of this matrix? (Note: Use Matlab's rank function.)
5. Apply SVD to your initial estimate of F.
6. What are the singular values of this estimate? Are the relative magnitudes about what you'd expect? Explain why.
7. Set the smallest singular value to zero and reconstitute a new estimate for F from the SVD by multiplying the resulting factors:
   

   F=UWVT
8. What is the rank of your adjusted estimate of the fundamental matrix F?

D. Visualizing Epipoles

Recall the equation for the right image epipolar line coefficients can be given by

uR=F - p   L

for an augmented point from the left image. The epipolar line in the right image is then expressed by

- p   T R  uR=0,

or

yR=−(u1xR+u3)/u2.

You can also do the reverse: give the right point to provide coefficients (this time in a row vector) for a line in the left image.

1. Plot (i.e., using a +) a new point (not one you used to construct the system matrix) in the left image and its corresponding epipolar line in the right image.
   Hint: Use the first and last column of the image as two x coordinates and calculate the corresponding y coordinates of these points for plotting.
2. Plot another new point in the right image and its corresponding epipolar line in the left image.
3. Do the points you selected in one image appear along the epipolar line in the other image? If not, how far away are they from one another? That, is what is the error?
4. Using the same method as D.1, plot the epipolar lines in the right image for all the points from the left image used in your system matrix.
5. Do your lines intersect? Where? What can you conclude about the epipolar geometry?
6. Can you determine whether these two images were taken at the same time? Why or why not?

Extra Credit / Project Starter Idea

Use your MOPS-based feature matching code to automatically find at least 8 point correspondences (you can use more in constructing the system matrix). Include your your existing kpfeat.m and kpdet.m and any other relevant functions in your submission and use these in a new procedure that takes two images as input and returns the fundamental matrix estimate. Full extra credit will be given for those solutions demonstrated to work correctly in a variety of situations; you may need to further tighten your detection/matching parameters so that you ensure you get at least 8 matches and that all are correct.

Acknowledgments

The stereo images were acquired by Jerod Weinman in his research lab and are released into the public domain.