1. Image Processing IB Paper 8 – Part A
Ognjen Arandjelović

2. – Image Essentials, Sampling –

3. Image Processing Paradigm
We can think of image processing as a black box that takes an input image and produces an output image.
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Feature extraction, for example, does not fall under the umbrella of image processing.

4. Image Processing
Image processing deals with computer-based manipulation of digital images. These include:
geometric operations (various morphs)
brightness and contrast correction
colour enhancement,
segmentation,
denoising.
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

5. Image Processing Applications
Fingerprint enhancement for analysis and recognition:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Original fingerprint image
After denoising and contrast enhancement
After binarization and morphological processing

6. Image Processing Applications
Enhancement of CCTV footage:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Still from original CCTV footage
After brightness and contrast enhancement
After denoising

7. Image Processing Applications
Achieving illumination and pose invariance for automatic face recognition:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
After face detection and cropping
After adaptive high-pass filtering
Still from original authentication video

8. Discretization
An inherent cause of information loss when representing physical signals on a computer:
Face geometry
Two types:
Level / numerical
It is impossible to represent infinite numbers (e.g. Pi) using a finite amount of storage space
Temporal or spatial
It is impossible to sample a signal at infinite frequency or at infinite number of locations
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

9. Discretization Under a Magnifying Glass
An example on an image of a face:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Pixels = spatial discretization
Values = numerical discretization

10. Numerical Quantization
The luminance of each pixel can be represented only with finite precision. This loss of information is called quantization noise.
Face geometry
We can compute the average quantization noise energy per pixel:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Minimal luminance difference that can be represented

11. Numerical Quantization – An Example
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

12. Spatial Discretization – An Example
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

13. 1D Sampling
In 1D, we model sampling of a continuous function as multiplication by a train of delta functions:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

14. 1D Sampling – Frequency Domain
Multiplication in the spatial domain goes into convolution in the frequency domain:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
A train of delta functions
Fourier spectrum of the original 1D signal
Fourier spectrum of the sampled 1D signal

15. 2D Sampling
Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
A comb of delta functions
Image as a 2D function / surface

16. 2D Sampling – Frequency Domain
Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
A comb of delta functions
Fourier spectrum of the sampled 2D signal
Fourier spectrum of a 2D signal

17. Geometric Transformations
Geometric transformations warp images without (in principle) changing the value of the corresponding 2D signal. Examples include:
Scaling
Cropping
Rotation
Arbitrary morphs
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

18. Geometric Transformations – Apps
Image registration for face recognition (correction of mild pose variations):
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

19. Geometric Transformations – Apps
Image warping for mosaicing of photographs:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
How to perform seamless concatenation of images?

20. Geometric Transformations – Main Idea
All geometric transformations of images at their core concern the problem of resampling.
As the original (not sampled) 2D signal cannot be accessed, we need to first reconstruct it from the set of samples we have i.e. pixels.
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

21. Signal Reconstruction
Remember that in the case of a band-limited signal, we can obtain a perfect reconstruction using the ideal low-pass filter:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Frequency spectrum of the sampled signal
Ideal low-pass filter
Perfectly reconstructed signal
What is the problem with this?

22. The Ideal Low-Pass Filter
Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function):
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Fourier transform
Ideal low-pass filter
The sinc function
N.B.

23. The Ideal Low-Pass Filter in 2D
Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function):
Fourier transform
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Ideal 2D low-pass filter
The 2D sinc function

24. Reconstruction in Spatial Domain
Using duality, we can see that the original signal can be reconstructed by convolving its sampled version with a sinc function:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

25. The Ideal LPF in the Spatial Domain
The reason why we cannot use the ideal LPF in image processing is of computational nature:
This value, for example depends on all samples!
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
The idea is to use something more manageable, by making the new sample dependent only on its neighbourhood.

26. Linear Interpolation
In linear interpolation, the signal between two samples is approximated by a straight line:
Clearly not perfect
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
We can thus say that the constraint is that the line should pass through the two sample points – there are 2 DOFs (a point on the line and the slope).

27. Linear Interpolation
We can then write an expression for the value of the function at an arbitrary location between two original samples:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Value at the new sample
Value of “pixel a”
Value of “pixel b”

28. Bi-Linear Interpolation
Bi-linear interpolation is simply a 2D extension of linear interpolation:
Original sampling locations
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
What is the value at this location?

29. Bi-Linear Interpolation
One way of thinking about bi-linear interpolation is as fitting of a plane between neighbouring non-collinear sampling points:
Original sampling locations
What is the value at this point?
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

30. Bi-Linear Interpolation
Alternatively, you can think of bi-linear interpolation as weighted average of the new sample’s neighbours:
Original sampling locations
What is the value at this point?
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

31. Cubic Interpolation
We have seen that linear interpolation is far from perfect. Cubic interpolation does better:
Much better!
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

32. Cubic Interpolation Constraints
In cubic interpolation we have the following constraints for segment between x2 and x3, given 4 consecutive samples x1, x2, x3, x4:
C0 Continuity: Value at x2 and x3 should be exact (as with linear interpolation)
C1 Continuity: Gradient at x2 should be (x3-x1) / 2 and at x3 should be (x4-x2) / 2
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

33. The Gradient Constraint
The see why the gradient at say x2 should be (x3-x1) / 2, consider 1st order Taylor series expansions around x2:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Now subtract (1) from (2):

34. Cubic Interpolation in 1D
The expression for cubic interpolation is thus more complicated:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

35. Bi-Cubic Interpolation
The expression for bi-cubic interpolation gets very messy, so we do not give it here.
The principle, however, is the same – you can think of it as cubic interpolation in the x direction, followed by cubic interpolation in the y direction.
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

36. Bi-Linear vs. Bi-Cubic Interpolation
A summary of key differences:
Cubic produces less smoothing of the signal than linear
Cubic is about 4 times more computationally demanding and is hence seldom used for resampling of large images
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

37. Impulse Response of Linear Interp.
We can analyse the performance of the two interpolators by looking at their impulse responses:
An impulse
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Impulse response of the linear interpolation process

38. Impulse Response of Linear Interp.
By taking the Fourier transform of impulse responses of the two interpolators, we can see how they affect different frequencies:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Frequency response of cubic interpolation is much flatter at higher frequencies – closer to the ideal LPF.

39. Image Rescaling
Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways.
New sampling locations (use interpolation)
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Original sampling locations
Image resizing as resampling – the original 5 х 5 image is resampled to 6 х 6.

40. Image Rescaling Example
To examine the effects of resampling, let us look at a small rectangular patch extracted from CCTV footage:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Magnified patch – 50 х 90 pixels

41. Image Rescaling Example
Let us now compare the results of bi-linear and bi-cubic interpolation-based magnification for a factor of 2:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Bi-cubic
Bi-linear
Note that bi-linear magnification produces a much smoother result.

42. Rescaling Comparison – An Example
Quantitative insight can be gained by looking at the image difference of two results:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
As expected from theory, the difference is mainly in the high-frequency content.

43. Downsampling Caveats
Remember that the original signal can be reconstructed from a set of samples using a LPF only if it is band-limited:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
A train of delta functions
Fourier spectrum of the original 1D signal
Fourier spectrum of the sampled 1D signal
No overlap of spectrum ‘replicas’ (this overlap is called aliasing)
It is crucial to ensure that the signal is band-limited before downsampling.

44. Downsampling Caveats – Aliasing
Let’s take a look at what happens with downsampling without considering the aliasing problem:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Downsampling
Original image (184 х 184 pixels)
Resulting image (40 х 40 pixels)
Note strange downsampling artefacts.

45. Downsampling Caveats – Aliasing
To ensure that high frequencies are suppressed, the original image should first be LP filtered:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Gaussian Smoothing
Original image (184 х 184 pixels)
Low-pass smoothed image

46. Downsampling Caveats – Aliasing
Now we can downsample as before, without worrying about aliasing:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Downsampling
Low-pass smoothed image
Resulting image (40 х 40 pixels)

47. Downsampling Caveats – Aliasing
Compare the results (or try switching anti-aliasing off in your Acrobat Reader!):
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Without LP filtering
With LP filtering

48. Image Rotation
Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways.
New sampling locations (use interpolation)
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Original sampling locations
Image rotation as resampling – the original 5 х 5 image is rotated ~30 degrees clockwise.

49. Image Rotation – New Samples
The main difference to rescaling is that the locations of new samples are now slightly more difficult to compute.
Recall that these are related to the original locations by the rotation matrix:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Where θ is the angle of rotation.

50. Image Rotation – Image Size
Note that the size of the resulting, rotated image is in general different than the size of the original image.
If the input image is of size H х W pixels, then the output image is of size H' х W' pixels, where:
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
The ceiling operator

51. Image Rotation – Image Size
Note that the size of the resulting, rotated image is in general different than the size of the original image.
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.

52. Image Rotation – An Example
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
Original fingerprint
(480 х 400 pixels)
Fingerprint rotated by 60 degrees
(586 х 615 pixels)

53. – That is All for Today –