Lab: Texture Synthesis

CSC 262 - Computer Vision - Weinman

Summary:

You will synthesize textures by matching histograms of steerable pyramid bands.

Deliverables

• The Matlab script used to make your comparisons and generate your report
• (5 points) Original and transformed noise images (A.4, A.8)
• (5 points) Sample image histogram (A.6)
• (5 points) Observations of initial matching (A.9)
• (5 points) Sample and synthesized image pyramid bands (B.5, B.6)
• (5 points) Sample and synthesized image band histograms (C.7)
• (5 points) Observations of band histograms (C.8)
• (5 points) Transformed/adjusted synthesized band image (C.11)
• (5 points) Comparison of band images (C.12)
• (5 points) Synthesized image after first iteration (D.5)
• (10 points) Observations of first iteration image (D.6)
• (10 points) Implementation of texturesyn (E)
• (15 points) Example synthesis results (E.4)
• (10 points) Analysis of synthesis results (E.5)
• (10 points) Professionalism of write-up

Extra Credit

• (10 points) Color version of the synthesis procedure with images

Preparation

If you are curious, you can have a look at the original Heeger and Bergen paper behind the method. Their algorithm is surprisingly simple! However, it does have its special conditions, which we may be inadvertently overlooking ... Several texture samples are on the MathLAN at

/home/weinman/courses/CSC262/images/texture

I suggest you start with carpet002-inca-100dpi-00.bmp as your sample image. Load it, converting it to grayscale doubles.

Exercises

Note that you will be assembling the results of your lab work into a procedure, so you are well advised to use variables, rather than hard-coded values, and choose meaningful names.

A. The Set-Up

1. The texture synthesis algorithm has several parameters:
   • Number of histogram bins
   • Number of pyramid scales
   • Number of pyramid orientations
   Name and choose some reasonable values for these. (Heeger and Bergen use 256 bins).
2. Use linspace to create a vector of evenly spaced points between 0 and 1 to represent the edges of your histogram bins. (Use the number you just selected above.)
3. We need to start the process with an image of random noise. For now, it can be the same size as your sample image. Use imnoise to add the default amount of Gaussian noise to a gray image (of doubles).
4. Display this starting image.
5. The Matlab histogram command histc takes a vector of data and a vector of bin edges

   H = histc(X, E);

   and returns the counts of all values in X that are between E(k) (inclusive) and E(k+1) (exclusive) in H(k). Use this to create a histogram of your sample image.
6. Plot your histogram.
7. The Matlab command histeq takes an image and a histogram

   Y = histeq(X, H);

   and adjusts the image X so that its histogram is approximately H. Use this to transform your input noise image so that it has the same histogram as the sample image.
8. Display this transformed image.
   Note: You may want to use subplot to combine this image with the original in A.4 for display in your report.
9. Does the transformed input image look more like the sample image than before? Does it actually look anything like the sample image otherwise?

B. Getting Pyramids and Bands

1. Use buildSFpyr to build a steerable pyramid of the sample image.
2. Use buildSFpyr to build a steerable pyramid of the appropriate size from your current synthesized texture image (the adjusted noise image from A.7).
3. Choose one of the pyramid bands to work with. (Remember that the second return value from buildSFpyr is a matrix containing the dimensions of each band.) Extract this band from your sample image using pyrBand.
4. Extract the same band from your (histogram matched) synthesized image.
5. Display the sample image's band (from B.3). (Don't forget to loosen your bounds!)
6. Display the synthesized image's band.

C. Adjusting Bands

We want to transform the synthesized image band so that its histogram matches that of the sample image band, just as we did in Part A. However, these images are now real-valued filter responses (or wavelet coefficients). Thus, they do not have the usual [0,1] range of normal images, which some image functions (such as histeq) expect. As a result, we must linearly re-adjust them so that they have this range. The provided (non-standard) function imadj takes an image and two ranges¹

Y = imadj(X,[low_in high_in],[low_out high_out]);

does a simple rescaling of the values in the input image X and so that the original values in between low_in and high_in are scaled to be between low_out and high_out.

1. Find the smallest value in your sample image band.
2. Find the largest value in your sample image band.
3. Create a version of the sample image band so that its range is mapped from the low and high values you just found to the more canonical [0,1] range.
4. Create a version of the synthesized image band so that its range is re-mapped with the same input range. (That is, do not find new low and high values.)
5. Use histc with your bins (from A.2) to find the histogram of the adjusted sample image band (from C.3).
6. Repeat the process to get the histogram of your adjusted synthesized image band.
7. Display these two histograms (you likely want to use subplot).
8. Compare/contrast the two histograms.
9. Transform the adjusted synthesized image band (from C.4) so that its histogram matches that of the adjusted sample image band (from C.5).
10. Adjust the transformed synthesized image band (from C.9) so that it is remapped from the [0,1] range back to the low and high values you originally remapped it from (in C.4).
11. Display the final synthesized band (from C.10).
12. Compare/contrast the band images from B.3, B.4, and C.10.

D. (Almost) Completing the Process

1. Write a for loop to apply steps B.3, B.4, and C.1-C.10 to every band in the pyramids.
2. At the end of your for loop, add a line that inserts each result (i.e., from C.10), as a single column vector, back into the steerable pyramid vector for the synthesized image. Use pyrBandIndices with the band number to get the appropriate indices into the vector.
3. Once your loop has completed, reconstruct the synthesized image using

   img = reconSFpyr(pyrValues, pyrDims);

   where the arguments are simply the values you got back from buildSFPyr.
4. Now the individual bands have been adjusted and transformed, but the total synthesized image needs to be re-transformed so that its histogram matches the sample image. Repeat the transformation of A.7 on your reconstructed image.
5. You've now completed a single iteration of texture synthesis! Display your newly adjusted, transformed image.
6. Compare your new image to before (i.e., 8).

E. Putting it Together

In this last bit, you will create a function

Y = texturesyn(X, N, M)

where X is a sample texture image that produces an N×M image Y synthesized from the sample.

1. Start by incorporating the work you did in parts A-D as the core of your procedure.
2. You will then need to wrap B.9-D.4 inside a while loop. Repeat the adjustment process until the successive difference between the synthesized images is sufficiently small (i.e., set a threshold on the average absolute difference):

   mean(abs(Iold(:)-Inew(:))

   You may also want to set a hard limit on the number of iterations so that it doesn't take forever, too.
3. You will likely need to explore to see what a good threshold might be. When do changes cease to be visually significant? Displaying your reconstruction within the loop could be helpful.
4. Display the original and synthesized results on a variety of the provided textures.
5. Comment on and try to explain the successes and failures of the method.

Extra Credit

The results of the algorithm are more impressive when they are in color. Make your procedure work on color images! You will need to repeat the processes for each color channel. However, remember that the algorithm assumes the bands are independent. RGB color channels are very non-independent, thus you will need to use a different color space whose component values are more independent of each other. You can try Y*Cr*Cb, which Matlab has built-in converters for. Be sure to include several synthesis comparisons with your enhanced procedure.

Acknowledgments

The texture samples are from the University of Oulu Texture Database "Outex."

---

Footnotes:

¹This function generalizes the Matlab command imadjust, which requires that its input and output ranges be subsets of [0,1].