Lab: Feature Matching

CSC 295 - Computer Vision - Weinman

Summary:

You will use your prior feature detection and descriptor functions as the basis for a matching algorithm that will allow you to stitch images assuming a translational transformation.

Due:

4/07

Deliverables

• The Matlab script(s) used to calculate values and and generate all figures
• (10 points) Sorted feature distances (B.7)
• (10 points) Feature distance observations (B.8)
• (15 points) Matched feature images and offset (C.4)
• (10 points) Estimated alignment (D.2)
• (10 points) Aligned images (D.4)
• (20 points) Alignment observations (D.5)
• (10 points) Professionalism of write-up

Preparation

/home/weinman/courses/CSC295/images/left.png

/home/weinman/courses/CSC295/images/right.png

/home/weinman/courses/CSC295/images/lectkpmatch1.png

/home/weinman/courses/CSC295/images/lectkpmatch1.png

Exercises

A. Matching Set-Up

1. Using your kpdet procedure, extract the keypoint locations from both images.
2. Using your kpfeat procedure, extract the keypoint descriptors from both images.
3. Use find to get the locations (row/col) of the features in both images.

B. Feature Matching

1. The function kpfeat should have returned an N×64 matrix of features for your reference image. Choose a feature index 1 < =k < =N from your reference image.
2. Extract the feature (the matrix row) as a vector and give it a variable name.
3. Recall that if a feature location indicated by kpdet could not be extracted in kpfeat, the entries should be NaN. The predicate procedure isnan determines whether a value is NaN. Use this to determine whether the feature you chose is "valid." If it is not, choose another value of k that is.
4. The procedure repmat repeats a matrix (or vector, or scalar) by tiling it as many times along each dimension as you specify. Use this to create a matrix from your extracted feature vector that is the same size as the matrix of features from the other (non-reference) image. That is, there should be a copy of your feature vector for each feature from the other image.
5. Use this "enlarged" (or repeated) version of your feature vector to calculate the Euclidean distance between your chosen feature and all the features of the other (non-reference) image. You now have two matrices of the same size, so you should be using vectorized operations.
   Note: You may need to tell Matlab which dimension you would like to calculate a sum along.
6. The resulting calculation should give you a vector of distances. Sort them in ascending order (the default), but use the version that returns two values from sort. This will provide you with not only the sorted values, but the original indices in the vector they came from.
7. Using bar, plot the distances of the top ten (closest) features. Save this figure.
8. Based on these distances, does your reference image feature appear to have a match in the other image? Discuss the quality of the match.
9. Calculate the estimated translation from the reference point location to the location of the best matching feature in the other imate. Note that the translation is simply the difference in locations
   

   tr = xr-yr tc = xc-yc

   where (x_r,x_c) are the row and column of the feature in the reference image and (y_r,y_c) are the row and column of the best matching feature in the other image.
   Note: If you used the sort command correctly, you know the index of the best matching feature in the descriptor matrix and thus its corresponding row and column as determined in A.3.

C. Visualizing Matching

1. Concatenated your reference image and the other image, either horizontally or vertically. They are both the same size, so either way is fine.
2. Display the concatenated image using imshow.
3. Use the command line to draw a line from the feature location in the reference image to the matching feature location in the other image. For instance, to draw a line from (row,col) pairs (1,3) to (2,4), you might do

   line([3 4],[1 2]);

   Note that line uses a different ordering of the parameters!
   Hint: In order to figure out the correct end pont, you wil need to use the size of the (reference) image along the dimension they were concatened as well as the reference feature location, and the offset calculated in B.9.
4. Save your image and report the offset.

D. Feature Matching Redux

1. Create a N×2 matrix of translation estimates for all the features in your reference image.
2. Add a for loop to iterate over all the descriptors from the reference image.
3. Inside your loop, add the command to break-out the current feature (as in B.2).
4. Next (inside the loop), if the current reference feature is invalid, discard the feature by setting the translation estimate for that feature to [NaN NaN] and continue the loop.
5. Create an array by replicating the current reference feature (as in B.4).
6. Measure the distance of the current reference feature to all the features in the other image (as in B.5).
7. Sort the distances and retain the (sorted) indices (as in B.6).
8. If the nearest-neighbor distance ratio exceeds a threshold (Brown et al. use 0.5), then discard the match by setting the translation estimate for the current feature to [NaN NaN] and continue the loop.
9. Finally, set the translation estimate of the matched features (as in B.9).

E. Alignment

1. Recall that the least squares solution to the translational transformation problem is simply to calculate the mean of the translations. Use this to find a single estimate of the best translation. Note that you will need to filter out all the NaN translations from your N×2 matrix.
2. What is the overall translation estimate? How much standard deviation is there in the estimate? What do you suppose this means for the potential quality of the aligned version of the images?
3. I have provided a procedure

   C = stitch(A, B, [Tr Tc]);

   that averages the images A and B under the translation given by Tr in the rows and Tc in the columns. Use this method and your estimated translation to create a stitched image.
4. Display and save your image.
5. How does the result look? Where does it work well? Where does it not work well? In both cases, why do you suppose that is so?
6. If you are not satisfied with the results, you may try
   1. getting a more robust estimate by using the median, rather than the mean, or
   2. tweaking parameters of your feature detection method (kpdet) or match method (i.e., the NNDR) so that more/fewer features are detected and/or more/fewer features are matched.
   If you employ any of these or other tweaks for improvement, fully describe and report on them in your write-up.