1. EE 7730: Image Analysis I Introduction

2. Bahadir K. Gunturk2 EE 7730 Dr. Bahadir K. Gunturk Office: EE 225 Email: bahadir@ece.lsu.edubahadir@ece.lsu.edu Tel: 8-5621 Office Hours: MW 2:40 – 4:30 Class Hours: MWF 1:40 – 2:30 (CEBA-3129)

3. Bahadir K. Gunturk3 EE 7730 We will learn the fundamentals of digital image processing, computer vision, and digital video processing Lecture slides, problems sets, solutions, study materials, etc. will be posted on the class website. [www.ece.lsu.edu/gunturk/EE7730] Textbook is not required. References:  Gonzalez/Woods, Digital Image Processing, Prentice-Hall, 2/e.  Forsyth/Ponce, Computer Vision: A Modern Approach, Prentice-Hall.  Duda, Hart, and Stork, Pattern Classification, John Wiley&Sons, 2001.  Tekalp, Digital Video Processing, 1995  Jain, Fundamentals of Digital Image Processing, Prentice-Hall.

4. Bahadir K. Gunturk4 Grading Policy Your grade will be based on  Problem Sets + Semester Project: 35%  Midterm: 30%  Final: 35% Problem Sets  Theoretical problems and MATLAB assignments  4-5 Problem Sets  Individually or in two-person teams Semester Project  Each student will give a 15 minute presentation

5. Bahadir K. Gunturk5 EE 7740 Image Analysis II Semester Project  Possible project topics will be provided in a month  Projects will be done individually  Projects will involve MATLAB or C/C++ implementation  Each student will give a 15 minute presentation at the end of the semester

6. Bahadir K. Gunturk6 EE 7740 Image Analysis II Image Analysis I - Outline  Digital image fundamentals  2D Fourier transform, sampling, Discrete Cosine Transfrom  Image enhancement  Human visual system and color image processing  Image restoration  Image compression  Image segmentation  Morphology  Introduction to digital video processing

7. Bahadir K. Gunturk7 Digital Image Acquisition Sensor array When photons strike, electron-hole pairs are generated on sensor sites. Electrons generated are collected over a certain period of time. The number of electrons are converted to pixel values. (Pixel is short for picture element.)

8. Bahadir K. Gunturk8 Digital Image Acquisition Two types of quantization: 1.There are finite number of pixels. (Spatial resolution) 2.The amplitude of pixel is represented by a finite number of bits. (Gray-scale resolution)

9. Bahadir K. Gunturk9 Digital Image Acquisition

10. Bahadir K. Gunturk10 Digital Image Acquisition 256x256 - Found on very cheap cameras, this resolution is so low that the picture quality is almost always unacceptable. This is 65,000 total pixels. 640x480 - This is the low end on most "real" cameras. This resolution is ideal for e-mailing pictures or posting pictures on a Web site. 1216x912 - This is a "megapixel" image size -- 1,109,000 total pixels -- good for printing pictures. 1600x1200 - With almost 2 million total pixels, this is "high resolution." You can print a 4x5 inch print taken at this resolution with the same quality that you would get from a photo lab. 2240x1680 - Found on 4 megapixel cameras -- the current standard -- this allows even larger printed photos, with good quality for prints up to 16x20 inches. 4064x2704 - A top-of-the-line digital camera with 11.1 megapixels takes pictures at this resolution. At this setting, you can create 13.5x9 inch prints with no loss of picture quality.

11. Bahadir K. Gunturk11 Matrix Representation of Images A digital image can be written as a matrix

12. Bahadir K. Gunturk12 Image Resolution

13. Bahadir K. Gunturk13 Bit Depth – Grayscale Resolution 8 bits7 bits 6 bits5 bits

14. Bahadir K. Gunturk14 Bit Depth – Grayscale Resolution 4 bits 3 bits 2 bits 1 bit

15. Bahadir K. Gunturk15 Digital Color Images

16. Bahadir K. Gunturk16 Video = vertical position = horizontal position = frame number ~24 frames per second.

17. Bahadir K. Gunturk17 Why do we process images? To facilitate their storage and transmission To prepare them for display or printing To enhance or restore them To extract information from them To hide information in them

18. Bahadir K. Gunturk18 Image Processing Example Image Restoration Original imageBlurredRestored by Wiener filter

19. Bahadir K. Gunturk19 Image Processing Example Noise Removal Noisy imageDenoised by Median filter

20. Bahadir K. Gunturk20 Image Processing Example Image Enhancement Histogram equalization

21. Bahadir K. Gunturk21 Image Processing Example Artifact Reduction in Digital Cameras Original sceneCaptured by a digital camera Processed to reduce artifacts

22. Bahadir K. Gunturk22 Image Processing Example Image Compression Original image 64 KB JPEG compressed 15 KB JPEG compressed 9 KB

23. Bahadir K. Gunturk23 Image Processing Example Object Segmentation “Rice” imageEdges detected using Canny filter

24. Bahadir K. Gunturk24 Image Processing Example Resolution Enhancement

25. Bahadir K. Gunturk25 Image Processing Example Watermarking Original image Hidden message Generate watermark Watermarked image Secret key

26. Bahadir K. Gunturk26 Image Processing Example Face Recognition Surveillance video Search in the database

27. Bahadir K. Gunturk27 Image Processing Example Fingerprint Matching

28. Bahadir K. Gunturk28 Image Processing Example Segmentation

29. Bahadir K. Gunturk29 Image Processing Example Texture Analysis and Synthesis Pattern repeated Computer generated Photo

30. Bahadir K. Gunturk30 Image Processing Example Face detection and tracking http://www-2.cs.cmu.edu/~har/faces.html

31. Bahadir K. Gunturk31 Image Processing Example Face Tracking

32. Bahadir K. Gunturk32 Image Processing Example Object Tracking

33. Bahadir K. Gunturk33 Image Processing Example Virtual Controls

34. Bahadir K. Gunturk34 Image Processing Example Visually Guided Surgery

35. Bahadir K. Gunturk35 Cameras First camera was invented in 16 th century. It used a pinhole to focus light rays onto a wall or translucent plate. Take a box, prick a small hole in one of its sides with a pin, and then replace the opposite side with a translucent plate. Place a candle on the pinhole side, you will see an inverted image of the candle on the translucent plate.

36. Bahadir K. Gunturk36 Perspective Projection Perspective projection equations

37. Bahadir K. Gunturk37 Pinhole Camera Model If the pinhole were really reduced to a point, exactly one light ray would pass through each point in the image plane. In reality, each point in the image place collects light from a cone of rays.

38. Bahadir K. Gunturk38 Pinhole Cameras Pinhole too big - many directions are averaged, blurring the image Pinhole too small - diffraction effects blur the image

39. Bahadir K. Gunturk39 Cameras With Lenses Most cameras are equipped with lenses. There are two main reasons for this:  To gather light. For an ideal pinhole, a single light ray would reach each point the image plane. Real pinholes have a finite size, so each point in the image plane is illuminated by a cone of light rays. The larger the hole, the wider the cone and the brighter the image => blurry pictures. Shrinking the pinhole produces sharper images, but reduces the amount of light and may introduce diffraction effects.  To keep the picture in sharp focus while gathering light from a large area.

40. Bahadir K. Gunturk40 Compound Lens Systems

41. Bahadir K. Gunturk41 Real Lenses Rays may not focus at a single point. Spherical aberration Spherical aberration can be eliminated completely by designing aspherical lenses.

42. Bahadir K. Gunturk42 Real Lenses Chromatic aberration The index of refraction is a function of wavelength. Light at different wavelengths follow different paths.

43. Bahadir K. Gunturk43 Real Lenses Chromatic Aberration

44. Bahadir K. Gunturk44 Real Lenses Special lens systems using two or more pieces of glass with different refractive indeces can reduce or eliminate this problem. However, not even these lens systems are completely perfect and still can lead to visible chromatic aberrations.

45. Bahadir K. Gunturk45 Real Lenses Barrel Distortion & Pincushion Distortion Stop (Aperture) Causes of distortion (normal) Chief ray

46. Bahadir K. Gunturk46 Real Lenses Barrel Distortion & Pincushion Distortion Distorted Corrected http://www.vanwalree.com/optics/distortion.html http://www.dpreview.com/learn/?/Image_Techniques/Barrel_Distortion_Correction_01.htm

47. Bahadir K. Gunturk47 Real Lenses Vignetting effect in a two-lens system. The shaded part of the beam never reaches the second lens. The brightness drop in the image perimeter.

48. Bahadir K. Gunturk48 Real Lenses Optical vignetting example. Left: f/1.4. Right: f/5.6. f-number focal length to diameter ratio

49. Bahadir K. Gunturk49 Real Lenses Long exposure time Short exposure time

50. Bahadir K. Gunturk50 Real Lenses Flare Hood may prevent flares

51. Bahadir K. Gunturk51 Real Lenses Flare

52. Bahadir K. Gunturk52 Compound Lens Systems http://www.dpreview.com/learn/?/glossary/ http://www.cartage.org.lb/en/themes/Sciences/Physics/Optics/Optical/Lens/Lens.htm http://www.vanwalree.com/optics.html

53. Bahadir K. Gunturk53 Digital Camera Pipeline Auto-exposure algorithms measure brightness over discrete scene regions to compensate for overexposed or underexposed areas by manipulating shutter speed and/or aperture size. The net goals here are to maintain relative contrast between different regions in the image and to achieve a good overall quality. (from Katz and Gentile)

54. Bahadir K. Gunturk54 Digital Camera Pipeline Auto-focus algorithms divide into two categories. Active methods use infrared or ultrasonic emitters/receivers to estimate the distance between the camera and the object being photographed. Passive methods, on the other hand, make focusing decisions based on the received image in the camera.

55. Bahadir K. Gunturk55 Digital Camera Pipeline Lens distortion correction This set of algorithms accounts for the physical properties of lenses that warp the output image compared to the actual scene the user is viewing. Different lenses can cause different distortions; for instance, wide-angle lenses create a "barrel distortion", while telephoto lenses create a "pincushion distortion“.

56. Bahadir K. Gunturk56 Digital Camera Pipeline Vignetting (shading distortion) reduces image brightness in the area around the lens. Chromatic aberration causes color fringes around an image. The media processor needs to mathematically transform the image in order to correct for these distortions.

57. Bahadir K. Gunturk57 Digital Camera Pipeline Sensor's output needs to be gamma-corrected to account for eventual display, as well as to compensate for nonlinearities in the sensor's capture response.

58. Bahadir K. Gunturk58 Digital Camera Pipeline Image stability compensation, or hand-shaking correction is another area of preprocessing. Here, the processor adjusts for the translational motion of the received image, often with the help of external transducers that relate the real-time motion profile of the sensor.

59. Bahadir K. Gunturk59 Digital Camera Pipeline White balance is another important stage of preprocessing. When we look at a scene, regardless of lighting conditions, our eyes tend to normalize everything to the same set of natural colors. For instance, an apple looks deep red to us whether we're indoors under fluorescent lighting, or outside in sunny weather. However, an image sensor's "perception" of color depends largely on lighting conditions, so it needs to map its acquired image to appear natural in its final output. This mapping can be done either manually or automatically.

60. Bahadir K. Gunturk60 Digital Camera Pipeline Demosaicking (Bayer interpolation) estimates missing color samples in single- chip cameras.

61. Bahadir K. Gunturk61 Digital Camera Pipeline In this stage, the interpolated RGB image is transformed to the targeted output color space (if not already in the right space). For compression or display to a television, this will usually involve an RGB  YCbCr matrix transformation, often with another gamma correction stage to accommodate the target display. The YCbCr outputs may also be chroma subsampled at this stage to the standard 4:2:2 format for color bandwidth reduction with little visual impact.

62. Bahadir K. Gunturk62 Digital Camera Pipeline Postprocessing In this phase, the image is perfected via a variety of filtering operations before being sent to the display and/or storage media. For instance, edge enhancement, pixel thresholding for noise reduction, and color-artifact removal are all common at this stage.

63. Bahadir K. Gunturk63 Digital Camera Pipeline Display / Compress / Store Once the image itself is ready for viewing, the image pipe branches off in two different directions. In the first, the postprocessed image is output to the target display, usually an integrated LCD screen (but sometimes an NTSC/PAL television monitor, in certain camera modes). In the second, the image is sent to the media processor's compression algorithm, where industry-standard compression techniques (JPEG, for instance) are applied before the picture is stored locally in some storage medium (e.g., Flash memory card).

64. Bahadir K. Gunturk64 Review: Linear Systems We define a system as a unit that converts an input function into an output function. System operator Independent variable

65. Bahadir K. Gunturk65 Linear Systems Then the system H is called a linear system. where f i (x) is an arbitrary input in the class of all inputs {f(x)}, and g i (x) is the corresponding output. Let If A linear system has the properties of additivity and homogeneity.

66. Bahadir K. Gunturk66 Linear Systems for all f i (x)  {f(x)} and for all x 0. The system H is called shift invariant if This means that offsetting the independent variable of the input by x 0 causes the same offset in the independent variable of the output. Hence, the input-output relationship remains the same.

67. Bahadir K. Gunturk67 Linear Systems The operator H is said to be causal, and hence the system described by H is a causal system, if there is no output before there is an input. In other words, A linear system H is said to be stable if its response to any bounded input is bounded. That is, if where K and c are constants.

68. Bahadir K. Gunturk68 Linear Systems (a)(a) a x  (x-a) A unit impulse function, denoted  (a), is defined by the expression

69. Bahadir K. Gunturk69 Linear Systems A unit impulse function, denoted  (a), is defined by the expression Then

70. Bahadir K. Gunturk70 Linear Systems is called the impulse response of H. The term From the previous slide It states that, if the response of H to a unit impulse [i.e., h(x,  )], is known, then response to any input f can be computed using the preceding integral. In other words, the response of a linear system is characterized completely by its impulse response.

71. Bahadir K. Gunturk71 Linear Systems and the integral becomes If H is a shift-invariant system, then This expression is called the convolution integral. It states that the response of a linear, fixed-parameter system is completely characterized by the convolution of the input with the system impulse response.

72. Bahadir K. Gunturk72 Linear Systems Convolution of two functions is defined as In the discrete case

73. Bahadir K. Gunturk73 Linear Systems is a linear filter. In the 2D discrete case

74. Bahadir K. Gunturk74 Example * =

75. Bahadir K. Gunturk75 Example * =

76. Bahadir K. Gunturk76 Try MATLAB f=imread(‘saturn.tif’); figure; imshow(f); [height,width]=size(f); f2=f(1:height/2,1:width/2); figure; imshow(f2); [height2,width2=size(f2); f3=double(f2)+30*rand(height2,width2); figure;imshow(uint8(f3)); h=[1 1 1 1; 1 1 1 1; 1 1 1 1; 1 1 1 1]/16; g=conv2(f3,h); figure;imshow(uint8(g));