Lab: Image Formation

CSC 262 - Computer Vision - Weinman

Summary:: We acquire some of our own images in a laboratory setting and use them to estimate the "noise" in the sensors.

Deliverables

For this and future labs you will submit the script you use to process the images, generate the data and your report as a single .m file. Note that you can create a Matlab script by using New -> Script. You may either add your commands directly to the file as you go, or copy them from your Command History. Be sure to clarify commands and sections with good documentation! (Omitted from the published report) For each lab, you will work through the instructions and be asked to include particular image or figure output along with discussions as follows.

Script containing the commands used to process the images, and generate the figures and your report as a single .m file.
(5 points) One of the stock images (A.2)
(10 points) The adjusted noise image (C.1)
(10 points) Observations on noise (C.2)
(15 points) Estimated noise plot (C.10)
(5 points) Noise gray level span (C.11)
(10 points) Average noise estimate (C.12)
(5 points) Maximum noise estimate (C.13).
(10 points) Professionalism of write-up

See Laboratory Write-Up and Submission for details.

Exercises

In this the lab, you will be using Matlab to make the calculations in the Algorithm EST_NOISE box from Trucco and Verri Section 2.2.3. One of the results you will produce will be a plot of the estimated acquisition noise for our cameras similar to Figure 2.11 in the text.

A set of pre-captured images located on the MathLAN in the following folder:

: /home/weinman/courses/CSC262/images/raw

A. Loading Images (5 minutes)

Start up Matlab. Change Matlab's workspace so that its working directory where these images are stored. (Use cd, just like in the shell.)
Whereas in the last lab you learned to use imread to load images, the "raw" images in this lab are nothing but pure bytes. Thus, Matlab does not know how to read them. Instead, you will use a different procedure called rawread to load them. For instance,

imgs = rawread('pic1.raw');

Load one of your images as above and display it using imshow.
Aside: If you ever want to see the code for a particular Matlab function, you can see it in the command window with the command

type rawread.m

or in the editor with

edit rawread.m
You may want to zoom in so you can see some of the individual pixels. Do you see strange plaid checkerboard patterns? The image you see is the readout (discretized to 0-255) from the sensors. These are color cameras and have an array of sensors that are more sensitive to different wavelengths of light (cf. Figure 2.31 in Szeliski). Zoom in (using the magnifying class) until you can see the individual raw pixels as larger squares. Do they look checkered? Make sure you understand why.
We've captured 10 images. In order to make it easier for us to let Matlab do the work of estimating the image noise, we are going to store all of these in a single 3D array, rather than separate 2D arrays.
Although Matlab does not require us to allocate space for an array beforehand, doing so can make your code run more quickly. We can force Matlab to pre-allocate 10 slices for our images by assigning one value to the last (lower right) pixel of the last image slice we want:

imgs(end,end,10) = 0;

Do this. Now examine the size of imgs. Is the result what you expect?
Your images can easily be loaded into this array with a for loop and a little creative string processing in Matlab. Actually, there are no strings in Matlab, only arrays of a char type. Thus, strings can be appended in just the same way that arrays can. Assuming you named them as suggested, the loop below will read the kth image into the corresponding slice of imgs using procedure num2str to convert from a numeric type to a char "string."

for k=1:10
imgs(:,:,k) = rawread(['pic' num2str(k) '.raw']);
end

Use this technique to load all of your images, but be sure you understand how it works.

B. Processing Images (10 minutes)

Our images are currently stored as bytes. Because we'll be doing math with them, we convert them into doubles in the range of 0-1 so we can still display them:

imgs = im2double(imgs);
Remember from your earlier Matlab lab that sum can add things up along a particular dimension. To calculate the standard deviation of each pixel, we first must calculate the mean of each pixel. Use sum (along the third dimension) to add up the values of the pixels in your image array along the slices and divide by the number of slices to give a "mean" image.

imgMean = sum( ...

The size of imgMean should be 960×1280. You can verify this using the workspace variables window, the command whos, or the command size.
To calculate the standard deviation, we need to take another sum, but we are adding the sum of (the square of) the difference between the mean image and one of the slices.
As you now know, to subtract two arrays they must be the of compatible size. Fortunately, Matlab automatically expands one of them for you:

imgDiff = imgs - imgMean;
Next we need to square the result. We are not taking a matrix power, but calculating the element-wise square of each value in the array. Recall that the .^ operator does this (carefully note the dot before the caret, which makes the operation element-wise).

imgDiffSqr = ...
Now that we have the squared difference, you can calculate the sum for of Eq. (2.17) using the same technique you used in finding imgMean, only this time note that the denominator is n-1. This quantity is called the variance.

imgVar = sum(imgDiffSqr, ...
Finally, you can take the sqrt of the result to give an "image" containing the standard deviation of each pixel. The result should once again be 960×1280.

imgStdDev = ...

C. Analyzing Noise (15 minutes)

You can display the deviation image using imshow. However, by default this procedure clips the values with black at 0 and white at 1. In order to show the lowest value in the image as black and the highest value as white, we can specify an empty display range as the second argument:

figure; % Open a new figure window
imshow(A,[])

Use this syntax to display your deviation image.
Do you notice more or less noise in certain regions of the image? Describe qualitatively what relationship there is, if any, between the image content and the amount of noise. Explain these phenomena to the best of your ability.
We'll try to get a better handle on the noise by looking at a curve along one row of pixels using the plot command, which expects a 1-D array of data to display. Recall that you can extract all the columns for an entire row using indexing like A(row,:). Using this, we can plot the middle row of the of the average image as follows

figure; % Open a new figure window
plot(imgMean(end/2,:));

Is the curve bumpier than you expect? This is because the brightness values plotted next to each other in the curve correspond to different color channels, thus registering different sensitivities to the light hitting the sensor.
We can extract just one of the color channels by looking only at the odd columns of one row. Write a command that plots only every other column in the same row of X, starting from 1. Does this look smoother?
If you didn't open a new figure, you may have noticed that plot overwrites whatever you had plotted previously. We are going to add two more curves to the plot you just made. In order to do this, we need to tell the figure to keep what was already there:

hold on;

Use this command to "hold on" to the current plot.
Next, plot the same brightness curve but with the standard deviation added to it. That is, if m is a brightness point of a particular pixel on the plot, you need to plot m+s, where s is the standard deviation of that pixel. To do this, you will extract the same array (or vector) as in the previous command and add to it a similar array extracted from the standard deviation image. (Make sure you are still only taking every other column of the data.)
Your plot may be clearer if you plot the added noise in a separate color.
Add to your plot the analogous curve for m-s (brightness minus standard deviation).
Use the magnifying glass to zoom into an interesting (there should be some change), but relatively stable (no wild swings) region of your new graph so that you can see all three curves.
Use the xlabel and ylabel commands to label the axes with something like 'Column' and 'Brightness'; don't forget a title!
Admire your graph. (This is the result you'll include in your report.)
Roughly how many discrete gray levels does the noise (twice the standard deviation) typically span? (Hint: Multiply by 255.)
What is the [estimate of the] average (over all the pixels) of the noise?
What is the [estimate of the] worst-case acquisition noise? (Hint: Matlab has a max function.)

Post Script

Once you publish your PDF, you may find that Matlab's warnings get included in the resulting document. This is unprofessional and undesirable, so we want to silence these. How can we do it?

Reconstructing the process from the matlab help, you find the last error by doing

: w = warning('query','last')

which in this case tells me it's

: w =
struct with fields:
identifier: 'images:initSize:adjustingMag' state: 'on'

So you could either suppress once before you generate your PDF with

: warning('off','last')

as suggested in the help, or the better solution would be to do it programmatically and specifically within your script:

: warning('off','images:initSize:adjustingMag')

Acknowledgments

The pre-captured raw images were created by Jerod Weinman and are licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. The exercise itself is inspired by "Algorithm EST_NOISE" (p. 32) of Trucco and Verri, Introductory Techniques for 3-D Computer Vision, Prentice Hall, 1998.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 International License.