1. Computer Vision Jia-Bin Huang, Virginia Tech
Image Pyramids and Applications
Computer Vision
Jia-Bin Huang, Virginia Tech
Computer Vision
Jia-Bin Huang, Virginia Tech
Golconda, René Magritte, 1953

2. Administrative stuffs
HW 0 posted – due Sept 10
HW 1 posted – due Sept 17

3. HW 0 – Basic image manipulation
1) Plot the R, G, B values along the scanline on the 250th row of the image. 2) Stack the R, G, B channels of the hokiebird image vertically. This will be an image with width of 600 pixels and height of 1116 pixels. 3) Load the input color image and swap its red and green color channels. 4) Convert the input color image to a grayscale image. 5) Take the R, G, B channels of the image. Compute an average over the three channels. Note that you may need to do the necessary typecasting (uint8 and double) to avoid overflow.

4. HW 0 – Basic image manipulation
6) Take the grayscale image, obtain the negative image (i.e., mapping 255 to 0 and 0 to 255). 7) First, crop the original hokie bird image into a squared image of size 372 x 372. Then, rotate the image by 90, 180, and 270 degrees and stack the four images (0, 90, 180, 270 degreess) horizontally. 8) Create another image with the same size as the hokie bird image. First, initialize this image as zero everywhere. Then, for each channel, set the pixel values as 255 when the corresponding pixel values in the hokie bird image are greater than ) Report the mean R, G, B values for those pixels marked by the mask in (8). 10) Take the grayscale image in (3). Create and initialize another image as all zeros. For each 5 x 5 window in the grayscale image, find out the maximum value and set the pixels with the maximum value in the 5x5 window as 255 in the new image. Some useful functions: title, subplot, imshow, mean, imread, imwrite, rgb2gray, find . Use help in MATLAB to learn more about these functions.

5. Previous class: Image Filtering
Linear filtering is sum of dot product at each position
Can smooth, sharpen, translate (among many other uses)
Gaussian filters
Low pass filters, separability, variance
Attend to details:
filter size, extrapolation, cropping
Applications:
Texture representation
Noise models and nonlinear image filters 1

6. Previous class: Image Filtering
Sometimes it makes sense to think of images and filtering in the frequency domain
Fourier analysis
Can be faster to filter using FFT for large images (N logN vs. N2 for auto-correlation)
Images are mostly smooth
Basis for compression
Remember to low-pass before sampling

7. Spatial domain * =
FFT
FFT
Inverse FFT =
Frequency domain

8. Upsampling This image is too small for this screen:
How can we make it 10 times as big?
Simplest approach:
repeat each row
and column 10 times
(“Nearest neighbor
interpolation”)

9. Image interpolation Recall how a digital image is formed
d = 1 in this
example 1 2 3 4 5
Recall how a digital image is formed
It is a discrete point-sampling of a continuous function
If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale
Adapted from: S. Seitz

10. Image interpolation Recall how a digital image is formed
d = 1 in this
example 1 2 3 4 5
Recall how a digital image is formed
It is a discrete point-sampling of a continuous function
If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale
Adapted from: S. Seitz

11. Image interpolation What if we don’t know ? Guess an approximation:1
d = 1 in this
example 1 2
2.5 3 4 5
What if we don’t know ?
Guess an approximation:
Can be done in a principled way: filtering
Convert to a continuous function:
Reconstruct by convolution with a reconstruction filter, h
Adapted from: S. Seitz

12. Image interpolation “Ideal” reconstruction
Nearest-neighbor interpolation
Linear interpolation
Gaussian reconstruction
Source: B. Curless

13. Reconstruction filters
What does the 2D version of this hat function look like?
performs
linear interpolation
(tent function) performs
bilinear interpolation
Often implemented without cross-correlation
E.g.,
Better filters give better resampled images
Bicubic is common choice
Cubic reconstruction filter

14. Image interpolation Original image: x 10
Nearest-neighbor interpolation
Bilinear interpolation
Bicubic interpolation

15. Image interpolation
Also used for resampling

16. Things to Remember
Sometimes it makes sense to think of images and filtering in the frequency domain
Fourier analysis
Can be faster to filter using FFT for large images (N logN vs. N2 for auto-correlation)
Images are mostly smooth
Basis for compression
Remember to low-pass before sampling

17. Today’s class Template matching Image Pyramids Compression
Introduction to HW1

18. Template matching Goal: find in image
Main challenge: What is a good similarity or distance measure between two patches? D() , )
Correlation
Zero-mean correlation
Sum Square Difference
Normalized Cross Correlation

19. Matching with filters Goal: find in image
Method 0: filter the image with eye patch
g = filter
f = image
Problem: response is stronger for higher intensity
What went wrong?
Input
Filtered Image

20. Matching with filters Goal: find in image
Method 1: filter the image with zero-mean eye
mean of template g
True detections
Likes bright pixels where filters are above average, dark pixels where filters are below average. Problems: response is sensitive to gain/contrast, pixels in filter that are near the mean have little effect, does not require pixel values in image to be near or proportional to values in filter.
False detections
Input
Filtered Image (scaled)
Thresholded Image

21. Matching with filters Goal: find in image
Method 2: Sum of squared differences (SSD)
True detections
Problem: SSD sensitive to average intensity
Input
1- sqrt(SSD)
Thresholded Image

22. Matching with filters Can SSD be implemented with linear filters?
Constant
Filtering with g
Filtering with box filter

23. What’s the potential downside of SSD?
Matching with filters
Goal: find in image
Method 2: SSD
What’s the potential downside of SSD?
Problem: SSD sensitive to average intensity
Input
1- sqrt(SSD)

24. Matching with filters Goal: find in image
Method 3: Normalized cross-correlation
mean template
mean image patch
Invariant to mean and scale of intensity
Matlab: normxcorr2(template, im)

25. Matching with filters Goal: find in image
Method 3: Normalized cross-correlation
True detections
Input
Thresholded Image
Normalized X-Correlation

26. Matching with filters Goal: find in image
Method 3: Normalized cross-correlation
True detections
Input
Thresholded Image
Normalized X-Correlation

27. Q: What is the best method to use?
A: Depends
Zero-mean filter: fastest but not a great matcher
SSD: next fastest, sensitive to overall intensity
Normalized cross-correlation: slowest, invariant to local average intensity and contrast

28. 2-mins break

29. Q: What if we want to find larger or smaller eyes?
A: Image Pyramid

30. Low-Pass Filtered Image
Review of Sampling
Gaussian
Filter
Sub-sample
Image
Low-Pass Filtered Image
Low-Res Image

31. Gaussian pyramid
Source: Forsyth

32. Template Matching with Image Pyramids
Input: Image, Template
Match template at current scale
Downsample image
In practice, scale step of 1.1 to 1.2
Repeat 1-2 until image is very small
Take responses above some threshold, perhaps with non-maxima suppression

33. Laplacian filter unit impulse Gaussian Laplacian of Gaussian
Source: Lazebnik

34. Laplacian pyramid
Source: Forsyth

35. Creating the Gaussian/Laplacian Pyramid
Image = G1
Smooth, then downsample G2
Downsample (Smooth(G1))
Downsample (Smooth(G2)) G3 …
GN = LN
G1 - Smooth(Upsample(G2)) L1 L2 L3
G3 - Smooth(Upsample(G4))
G2 - Smooth(Upsample(G3))
Use same filter for smoothing in each step (e.g., Gaussian with 𝜎=2)
Downsample/upsample with “nearest” interpolation

36. Hybrid Image in Laplacian Pyramid
High frequency  Low frequency

37. Reconstructing image from Laplacian pyramid
L1 + Smooth(Upsample(G2))
G2 = L2 + Smooth(Upsample(G3))
G3 = L3 + Smooth(Upsample(L4)) L4 L1 L2 L3
Use same filter for smoothing as in desconstruction
Upsample with “nearest” interpolation
Reconstruction will be lossless

38. Major uses of image pyramids
Object detection
Scale search
Features
Detecting stable interest points
Course-to-fine registration
Compression

42. Coarse-to-fine Image Registration
Compute Gaussian pyramid
Align with coarse pyramid
Successively align with finer pyramids
Search smaller range
Why is this faster?
Are we guaranteed to get the same result?

43. Applications: Pyramid Blending

44. Applications: Pyramid Blending

45. Pyramid Blending At low frequencies, blend slowly
At high frequencies, blend quickly 1 1 1
Left pyramid
Right pyramid
blend

46. Image representation Pixels:
great for spatial resolution, poor access to frequency
Fourier transform:
great for frequency, not for spatial info
Pyramids/filter banks:
balance between spatial and frequency information

47. Compression
How is it that a 4MP image (12000KB) can be compressed to 400KB without a noticeable change?

48. Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
Slides: Efros

49. Using DCT in JPEG
The first coefficient B(0,0) is the DC component, the average intensity
The top-left coeffs represent low frequencies, the bottom right – high frequencies

50. Image compression using DCT
Quantize
More coarsely for high frequencies (which also tend to have smaller values)
Many quantized high frequency values will be zero
Encode
Can decode with inverse dct
Filter responses
Quantization table
Quantized values

51. JPEG Compression Summary
Convert image to YCrCb
Subsample color by factor of 2
People have bad resolution for color
Split into blocks (8x8, typically), subtract 128
For each block
Compute DCT coefficients
Coarsely quantize
Many high frequency components will become zero
Encode (e.g., with Huffman coding)

52. Lossless compression (PNG)
Predict that a pixel’s value based on its upper-left neighborhood
Store difference of predicted and actual value
Pkzip it (DEFLATE algorithm)

53. Three views of image filtering
Image filters in spatial domain
Filter is a mathematical operation on values of each patch
Smoothing, sharpening, measuring texture
Image filters in the frequency domain
Filtering is a way to modify the frequencies of images
Denoising, sampling, image compression
Templates and Image Pyramids
Filtering is a way to match a template to the image
Detection, coarse-to-fine registration

54. HW 1 – Hybrid Image Hybrid image =
Low-Freq( Image A ) + Hi-Freq( Image B )

55. HW 1 – Image Pyramid

56. HW 1 – Edge Detection Derivative of Gaussian filters x-direction
y-direction
Derivative of Gaussian filters

57. Things to remember Template matching (SSD or Normxcorr2)
SSD can be done with linear filters, is sensitive to overall intensity
Gaussian pyramid
Coarse-to-fine search, multi-scale detection
Laplacian pyramid
More compact image representation
Can be used for compositing in graphics
Compression
In JPEG, coarsely quantize high frequencies

58. Thank you
See you this Thursday
Next class:
Edge detection