1. Vision, Video and Virtual Reality Feature Extraction Lecture 7 Image Enhancement CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html

2. Vision, Video and Virtual Reality What are Image Features? n Local, meaningful, detectable parts of the image.

3. Vision, Video and Virtual Reality More Color Woes Squares with dots in them are the same color

4. Vision, Video and Virtual Reality Topics n Image Enhancement l Brightness mapping l Contrast stretching/enhancement l Histogram modification l Noise Reduction l ……... n Mathematical Techniques l Convolution l Gaussian Filtering n Edge and Line Detection and Extraction n Region Segmentation n Contour Extraction n Corner Detection

5. Vision, Video and Virtual Reality Image Enhancement n Goal: improve the ‘visual quality’ of the image l for human viewing l for subsequent processing n Two typical methods l spatial domain techniques.... u operate directly on image pixels l frequency domain techniques.... u operate on the Fourier transform of the image n No general theory of ‘visual quality’ l General assumption: if it looks better, it is better l Often not a good assumption

6. Vision, Video and Virtual Reality Spatial Domain Methods n Transformation T l point - pixel to pixel l area - local area to pixel l global - entire image to pixel n Neighborhoods l typically rectangular l typically an odd size: 3x3, 5x5, etc l centered on pixel I(x,y) n Many IP algorithms rely on this basic notion T(I(x,y)) I(x,y)I’(x,y)neighborhood N I’(x,y) = T(I(x,y)) O=T(I)

7. Vision, Video and Virtual Reality Point Transforms: General Idea O = T(I) 0 255 INPUT OUTPUT Input pixel value, I, mapped to output pixel value, O, via transfer function T. Transfer Function T

8. Vision, Video and Virtual Reality Grayscale Transforms n Photoshop ‘adjust curve’ command Input gray value I(x,y) Output gray value I’(x,y)

9. Vision, Video and Virtual Reality Point Transforms: Brightness

10. Vision, Video and Virtual Reality Point Transforms:Thresholding n T is a point-to-point transformation l only information at I(x,y) used to generate I’(x,y) n Thresholding I max if I(x,y) > t I min if I(x,y) ≤ t I’(x,y) = t=89 t2550 0

11. Vision, Video and Virtual Reality Point Transforms: Linear Stretch 0 255 INPUT OUTPUT

12. Vision, Video and Virtual Reality Linear Scaling n Consider the case where the original image only utilizes a small subset of the full range of gray values: n New image uses full range of gray values. n What's F? {just the equation of the straight line}

13. Vision, Video and Virtual Reality Linear Scaling n F is the equation of the straight line going through the point (I min, 0) and (I max, K) n useful when the image gray values do not fill the available range. n Implement via lookup tables

14. Vision, Video and Virtual Reality Scaling Discrete Images n Have assumed a continuous grayscale. n What happens in the case of a discrete grayscale with K levels?

15. Vision, Video and Virtual Reality Examples Lighten Darken Emphasize Dark PixelsLike Photographic Solarization

16. Vision, Video and Virtual Reality Non-Linear Scaling: Power Law n  < 1 to enhance contrast in dark regions n  > 1 to enhance contrast in bright regions. O = I  00.51 0 0.5 1   

17. Vision, Video and Virtual Reality Square Root Transfer:  =.5

18. Vision, Video and Virtual Reality  =3.0

19. Vision, Video and Virtual Reality Examples n Technique can be applied to color images l same curve to all color bands l different curves to separate color bands:

20. Vision, Video and Virtual Reality Point Transforms:Thresholding n T is a point-to-point transformation l only information at I(x,y) used to generate I’(x,y) n Thresholding I max if I(x,y) > t I min if I(x,y) ≤ t I’(x,y) = t=89 t2550 0

21. Vision, Video and Virtual Reality Threshold Selection n Arbitrary selection l select visually n Use image histogram Threshold

22. Vision, Video and Virtual Reality Histograms n The image shows the spatial distribution of gray values. n The image histogram discards the spatial information and shows the relative frequency of occurrence of the gray values.

23. Vision, Video and Virtual Reality Image Histogram n The histogram typically plots the absolute pixel count as a function of gray value: For an image with dimensions M by N I max

24. Vision, Video and Virtual Reality Probability Interpretation n The graph of relative frequency of occurrence as a function of gray value is also called a histogram: n Interpreting the relative frequency histogram as a probability distribution, then:   P(I(x,y) = i) = H(i)/(MxN)

25. Vision, Video and Virtual Reality Cumulative Density Function n Interpreting the relative frequency histogram as a probability distribution, then: n Curve is called the cumulative distribution function Gray Value P(I(x,y) = i) = H(i)/(MxN)

26. Vision, Video and Virtual Reality Examples

27. Vision, Video and Virtual Reality Color Histograms

28. Vision, Video and Virtual Reality Histogram Equalization n Image histograms consist of peaks, valleys, and low plains n Peaks = many pixels concentrated in a few grey levels n Plains = small number of pixels distributed over a wider range of grey levels

29. Vision, Video and Virtual Reality Histogram Equalization n The goal is to modify the gray levels of an image so that the histogram of the modified image is flat. l Expand pixels in peaks over a wider range of gray-levels. l “Squeeze” low plains pixels into a narrower range of gray levels. n Utilizes all gray values equally n Example Histogram: l Note low utilization of small gray values

30. Vision, Video and Virtual Reality Desired Histogram n All gray levels are used equally. n Has a tendency to enhance contrast.

31. Vision, Video and Virtual Reality Brute Force How are the gray levels in the original image changed to produce the enhanced image? Method 1. Choose points randomly. Method 2. Choice depends on the gray levels of their neighboring points. Computationally expensive. Approximations.

32. Vision, Video and Virtual Reality Histogram Equalization Mapping from one set of grey-levels, , to a new set, . n Ideally, the number of pixels, N p, occupying each grey level should be: n To approximate this, apply the transform l Where CH is the cumulative histogram (see next slide) l j is the gray value in the source image l i is the gray value in the equalized image N p = M*N G i = MAX 0, round CH(j) NpNp

33. Vision, Video and Virtual Reality Example 800 600 400 200 0 0 1 2 3 4 5 6 8 ideal 800 600 400 200 0 G=8 MxN=2400 N p =300 jH(j)CH(j)i 01000 18002 27004 35006 41006 5 7 6 7 707 900 1600 2100 2200 2300 2400 CH(j) =  H(i) i=0 j 0 1 2 3 4 5 6 8

34. Vision, Video and Virtual Reality Example

35. Vision, Video and Virtual Reality Comparison Original  >1 Histogram equalization

36. Vision, Video and Virtual Reality Why? n Given MxN input image I with gray scale p 0 ….p k and histogram H(p). n Desire output image O with gray scale q 0 ….q k and uniform histogram G(q) n Treat the histograms as a discrete probability density function. Then monotonic transfer function m = T(n) implies: n The sums can be interpreted as discrete distribution functions. From Image Processing, Analysis, and Machine Vision, Sonka et al.  G(q i ) =  H(p j ) i=0 j=0 m n

37. Vision, Video and Virtual Reality Why? continued n The histogram entries in G must be: (because G is uniform) n Plug this into previous equation to get: n Now translate into continuous domain (can’t really get uniform distributions in the discrete domain) MxN q k - q 0 G(i) =  =  H(p i ) i=0 m n MxN q k - q 0 MxN q k - q 0 =  H(s)ds p0p0 pnpn q0q0 qmqm ds MN(q m -q 0 ) q k - q 0

38. Vision, Video and Virtual Reality Why? continued n Solve last equation for q to get: n Map back into discrete domain to get: n Leads to an algorithm….. MN q k - q 0 H(s)ds + q 0 p0p0 pnpn q m = T(p) = MN q k - q 0  H(i) + q 0 q m = T(p) = i=p 0 pnpn

39. Vision, Video and Virtual Reality Histogram Equalization Algorithm n For an NxM image of G gray levels, say 0-255 l Create image histogram l For cumulative image histogram H c n Set n Rescan input image and write new output image by setting T(p) = round ( H c (p)) G-1 NM g q = T(g p )

40. Vision, Video and Virtual Reality Observations n Acquisition process degrades image n Brightness and contrast enhancement implemented by pixel operations n No one algorithm universally useful n  > 1 enhances contrast in bright images n  < 1 enhances contrast in dark images n Transfer function for histogram equalisation proportional to cumulative histogram

41. Vision, Video and Virtual Reality Noise Reduction n What is noise? n How is noise reduction performed? l Noise reduction from first principles l Neighbourhood operators u linear filters (low pass, high pass) u non-linear filters (median) image + noise = ‘grainy’ image

42. Vision, Video and Virtual Reality Noise n Plot of image brightness. n Noise is additive. n Noise fluctuations are rapid l high frequency. Image Noise Image + Noise

43. Vision, Video and Virtual Reality Sources of Noise n Sources of noise = CCD chip. n Electronic signal fluctuations in detector. n Caused by thermal energy. n Worse for infra-red sensors. n Electronics n Transmission Radiation from the long wavelength IR band is used in most infrared imaging applications

44. Vision, Video and Virtual Reality Noise Model 22  n Plot noise histogram n Typical noise distribution is normal or Gaussian n Mean(noise)  = 0 n Standard deviation   (x) = exp - 1 2   1 2 (x-   2

45. Vision, Video and Virtual Reality Effect of  n Integral under curve is 1  =10  =20  =30

46. Vision, Video and Virtual Reality Two Dimensional Gaussian

47. Vision, Video and Virtual Reality Noise n Noise varies above and below uncorrupted image. Image + Noise

48. Vision, Video and Virtual Reality Noise Reduction - 1 n How do we reduce noise? n Consider a uniform 1-d image and add noise. n Focus on a pixel neighbourhood. n Averaging the three values should result in a value closer to original uncorrupted data n Especially if gaussian mean is zero A i-1 A i A i+1 Ci Ci

49. Vision, Video and Virtual Reality Noise Reduction - 1 n Averaging ‘smoothes’ the noise fluctuations. n Consider the next pixel A i+1 n Repeat for remainder of pixels. A i-1 A i A i+1 A i+2 C i+1

50. Vision, Video and Virtual Reality Neighborhood Operations n All pixels can be averaged by convolving 1-d image A with mask B to give enhanced image C. n Weights of B must equal one when added together. l Why? C = A * B B = [ B1 B2 B3 ] C i = A i-1  B 1 + A i  B 2 + A i+1  B 3 B = [1 1 1] C i = 1 3 A i-1 + A i + A i+1 3

51. Vision, Video and Virtual Reality 2D Analog of 1D Convolution n Consider the problem of blurring a 2D image n Here’s the image and a blurred version: n How would we do this? l What’s the 2D analog of 1D convolution?

52. Vision, Video and Virtual Reality 2D Blurring Kernel n C = A * B n B becomes 1 1 1 B = 1 9

53. Vision, Video and Virtual Reality Convolution n Extremely important concept in computer vision, image processing, signal processing, etc. n Lots of related mathematics (we won’t do) n General idea: reduce a filtering operation to the repeated application of a mask (or filter kernel) to the image l Kernel can be thought of as an NxN image l N is usually odd so kernel has a central pixel n In practice l (flip kernel) l Align kernel center pixel with an image pixel l Pointwise multiply each kernel pixel value with corresponding image pixel value and add results l Resulting sum is normalized by kernel weight l Result is the value of the pixel centered on the kernel

54. Vision, Video and Virtual Reality Image Kernel Result Image Example Kernel is aligned with pixel in image, multiplicative sum is computed, normalized, and stored in result image. n Process is repeated across image. What happens when kernel is near edge of input image?

55. Vision, Video and Virtual Reality Border Problem n missing samples are zero n missing samples are gray n copying last lines n reflected indexing (mirror) n circular indexing (periodic) n truncation of the result

56. Vision, Video and Virtual Reality Image size = M 1   N 1 Mask size = M 2   N 2 Convolution size = M 1 - M 2 +1   N 1 -N 2 +1 N1N1 N2N2 N 1 -N 2 +1 Typical Mask sizes = 3  3, 5  5, 7  7, 9  9, 11  11 What is the convolved image size for a 128   128 image and 7  7 mask? Convolution Size

57. Vision, Video and Virtual Reality Convolution = I M *

58. Vision, Video and Virtual Reality Properties of Convolution n Commutative: f 1 (t) * f 2 (t) = f 2 (t) * f 1 (t) n Distributive: f 1 (t) * [f 2 (t) + f 3 (t)] = f 1 (t) * f 2 (t) + f 1 (t) * f 3 (t) n Associative: f 1 (t) * [f 2 (t) * f 3 (t)] = [f 1 (t) * f 2 (t)] * f 3 (t) n Shift: if f 1 (t) * f 2 (t) = c(t), then l f 1 (t) * f 2 (t-T) = f 1 (t-T) * f 2 (t) = c(t-T) Convolution with impulse: f(t) *  (t) = f(t) Convolution with shifted impulse: f(t) *  (t-T) = f(t-T)

59. Vision, Video and Virtual Reality Noise Reduction - 1 Uncorrupted Image Image + Noise Image + Noise - Blurred

60. Vision, Video and Virtual Reality Noise Reduction - 1 n Technique relies on high frequency noise fluctuations being ‘blocked’ by filter. Hence, low-pass filter. n Fine detail in image may also be smoothed. n Balance between keeping image fine detail and reducing noise.

61. Vision, Video and Virtual Reality Noise Reduction - 1 n Saturn image coarse detail n Boat image contains fine detail n Noise reduced but fine detail also smoothed

62. Vision, Video and Virtual Reality hmmmmm….. ImageBlurred Image -=

63. Vision, Video and Virtual Reality Noise Reduction - 2: Median Filter n Nonlinear Filter n Compute median of points in neighborhood n Has a tendency to blur detail less than averaging n Works very well for ‘shot’ or ‘salt and pepper’ noise OriginalLow-passMedian

64. Vision, Video and Virtual Reality Noise Reduction - 2 Low-passMedian Low-pass: fine detail smoothed by averaging Median: fine detail passed by filter

65. Vision, Video and Virtual Reality Edge Preserving Smoothing n Smooth (average, blur) an image without disturbing l Sharpness or l position Of the edges n Nagoa-Maysuyama Filter n Kuwahara Filter n Anisotropic Diffusion Filtering (Perona & Malik,.....) n Spatially variant smoothing

66. Vision, Video and Virtual Reality Nagao-Matsuyama Filter n Calculate the variance within nine subwindows of a 5x5 moving window n Output value is the mean of the subwindow with the lowest variance n Nine subwindows used:

67. Vision, Video and Virtual Reality Kuwahara Filter n Principle: l divide filter mask into four regions (a, b, c, d). l In each compute the mean brightness and the variance l The output value of the center pixel (abcd) in the window is the mean value of that region that has the smallest variance.

68. Vision, Video and Virtual Reality Kuwahara Filter Example Original Median (1 iteration) Median (10 iterations) Kuwahara

69. Vision, Video and Virtual Reality Anisotropic Diffusion Filtering Note: Original Perona-Malik diffusion process is NOT anisotropic, although they erroneously said it was.

70. Vision, Video and Virtual Reality Example: Perona-Malik n In general: l Anisotropic diffusion estimates an image from a noisy image l Useful for restoration, etc. l Only shown smoothing examples l Read the literature

71. Vision, Video and Virtual Reality Observations on Enhancement n Set of general techniques n Varied goals: l enhance perceptual aspects of an image for human observation l preprocessing to aid a vision system achieve its goal n Techniques tend to be simple, ad-hoc, and qualitative n Not universally applicable l results depend on image characteristics l determined by interactive experimentation n Can be embedded in larger vision system l selection must be done carefully l some techniques introduce artifacts into the image data n Developing specialized techniques for specific applications can be tedious and frustrating.

72. Vision, Video and Virtual Reality Summary n What is noise? l Gaussian distribution n Noise reduction l first principles n Neighbourhood l low-pass l median n Averaging pixels corrupted by noise cancels out the noise. n Low-pass can blur image. n Median may retain fine image detail that may be smoothed by averaging. Conclusion