Lab: Feature Descriptors

CSC 295 - Computer Vision - Weinman

Summary:

You will implement a version of Brown et al.'s multi-scale oriented patches (MOPS), but without the multiscale and oriented bits.

Due:

Fri 3/19

Deliverables

• The Matlab script used to make your comparisons and generate all figures
• (10 points) Keypoint location reference image (B.9)
• (20 points) Small, large, and small normalized patch images (B.6, B.7, C.3)
• (10 points) Full-size and downsampled patch observations (B.8)
• (10 points) Patch normalization observations (C.4)
• (20 points) Descriptor extraction function kpfeat (D)
• (10 points) Professionalism of write-up

Preparation

/home/weinman/courses/CSC295/images/kpdet1.tif

Exercises

A. Processing for Descriptors

1. Before extracting the feature descriptors, you need to find the features. Run your kpdet procedure on the image to get the matrix of detected feature locations.
2. Use find to locate the rows/columns where the feature keypoints were detected.
3. The MOPS descriptors are 8×8 patches, so you will need a place to (potentially) put them all. Initialize an N×64 vector, where N is the number of detected keypoints.
   Note: Your input image may change. Do not hard-code a number for N. Instead, calculate it from your previous results.
4. The MOPS descriptor requires a very coarse representation of the image. To safely get such a coarse image, we blur and then downsample Create a Gaussian kernel with a scale (standard deviation) of 3 pixels (so the variance is 3²).
5. Apply the Gaussian kernel along both dimensions to blur the input image, and ask for a result the same size as the original image.
6. Downsample the image by a factor of 5 (That is, take every fifth row and column).

B. Extracting Patches

1. You have detected N key points and produced vectors containing their row and column in A.2. Assign a variable k to a value between 1 and N to be one of these keypoints. This will also correspond the row of the descriptor matrix you created in A.3.
   Eventually k will be a loop variable, but you may pick its value arbitrarily for now.
2. We need to find the appropriate 8×8 patch in the downsized image for the keypoint detected. To do this, you'll first need to find the index in the downsized image where the patch is centered. Find a scaled location by dividing the row and column (using the index variable k) of the feature by the subsampling factor.
   Note: since you need image/matrix indices, the result must be made into an integer.
3. The scaled row and column you calculated in the previous step indicate the center location of your patch. Subtract an offset from these so that you have the upper-left corner of the 8×8 patch. (It may help to draw a grid.)
   Reminder: You are doing this in the downsized image.
4. Calculate (and save) the lower-right corner of your 8×8 patch in the downsized image.
5. Using the coordinates you calculated, extract the patch from your downsized image. Verify that it is 8×8. If not, you should adjust your math.
   Note: If you get errors about your subscript indices (that they must be positive or exceed the matrix dimensions) choose a different value for k and try again. We'll fix that error in a moment.
6. Display your image patch. The following sequence of commands are a handy way to visualize very small images

   imagesc(img,[0 1]); % Display the image with black 0 and white 1

   colormap(gray);     % Render in grayscale

   axis equal off;     % Use square pixels and turn off borders

   Save your image figure.
7. Extract and display the corresponding 8·5×8·5 patch from the original image. Save this original patch.
8. How do the two patches compare?
9. For reference, display your original image with a plus where the feature keypoint (patch center) location is. Save this result.

C. Patch Processing

1. To normalize the bias, calculate a new version of the 8×8 patch by subtracting the mean of all the pixels in the patch. (Remember the mean procedure expects a vector!)
2. To normalize the gain, calculate another version of the patch by dividing the bias-normalized version by its variance. (The var procedure also expects a vector.)
3. Display your normalized patch. (If you omit the bounds for imagesc, it will automatically use the lowest and highest values in the image matrix as black and white, respectively). Save this image.
4. How does the normalized patch compare visually to its corresponding un-normalized version?

D. Processing Keypoints

1. Open a file called kpfeat.m in your Matlab editor of choice.
2. Add a declaration line to your function so that it takes two parameters, an image and its corresponding matrix of feature keypoint detections (i.e., the output of kpdet), and returns one value, the matrix of feature descriptors.
3. Add your initial code for processing the image to the file (e.g. A.2-A.6).
4. Add a for loop over all the N keypoints given by the second argument. Within your loop you should apply the steps from parts C and D to extract and normalize the patch.
   Note: You do not need visualize anything here or extract the larger patch.
5. Set the kth row of your N×64 matrix to the entries in your normalized patch. For instance,

   features(k,:) = norm_patch(:);
6. There is a possibility that the region you are trying to extract the patch from may fall outside the bounds of the image. Add a check to make sure that the upper-left corner and lower-right corner of the image all fall within bounds. If they fall outside of the bounds, set the entries in the kth row of your feature matrix to the Matlab value NaN (not a number) and use the keyword continue to advance the loop to the next iteration. We'll keep these entries around so that there is some agreement with the output of the kpdet function.
   Hint: The size command can tell you a variable's dimensions.
7. You should now have a procedure that extracts the feature descriptors from an image. Test it out! In the next lab, we'll implement a matcher.