1. 1 Machine Vision lecture 1 Thomas Moeslund Computer Vision and Media Technology lab. Aalborg University tbm@cvmt.dk

2. 2 Pose estimation –Pick and place applications –Bin-picking Machine vision applications Classification Quality control THOR HOF THOR Conveyor belt

3. 3 Machine Vision class Purpose: –Provide an overview of the different elements in a machine vision system

4. 4 Machine Vision class Topics –Acquisition of images: Light, camera, lens –Representation of digital images: pixels, colors –Processing of images filtering, segmentation –Analysis of images: feature extraction, pattern recognition –Your interests and needs?

5. 5 Plan for today Where does an image come from? Image definitions Camera types Image formation –Lens Creating light for machine vision ( other applications )

6. 6 Where does an image come from?

7. 7

8. 8 Charged coupled device CCD-chip

9. 9 Where does an image come from? Integration over time –Exposure time/shutter time DK: lukketid –Maximum charge Saturation Blooming Under exposed Over exposed Correct exposed

10. 10 Where does an image come from? Image elements, picture elements, pels, pixels

11. 11 Image definitions

12. 12 Digital Image Representation Image is seen as a discrete function f(x,y) as opposed to a continuous function x and y cannot take on any value! y x f(x,y) Origin

13. 13 Discrete image coordinate system y x f(x,y) Origin y x f(0,0) f(2,6)

14. 14 Digital Image Representation Pixel representation (bits) –One bit: {0,1} –One byte = eight bits –One pixel: one byte = eight bits = one number: [0,255] –Grey-scale, intensity, black/white: 8 bits = [0,255] –Binary image: 1 bit {0,1}. Black and white: visualized as: 8 bit {0,255} ( colors: Another lecture ) Video: 25 – 60 images per second. –Framerate: 25Hz – 60Hz –For example: 500 x 500 pixel at 50Hz => 1.25x10 7 bytes per second!!!!

15. 15 Digital Image Representation An image f(x,y) is represented as an Array Width = number of pixels in x-direction Height = number of pixels in y-direction Size (width x height, width > height) ROI = region of interest –To reduce the amount of data Width Height ROI

16. 16 Spatial Image Resolution: Resolution –The size of an area in a scene that is represented by one pixel in the image Different Resolutions are possible (256x256….16x16) Lower resolution leads to data reduction!

17. 17 Gray-level Resolution: Quantization Different gray-level resolutions: 256, 128, …, 2 Less gray-levels leads to data reduction. For 256, 128, 64 gray-levels: Difference hardly visible

18. 18 Camera types

19. 19 Interlaced vs. Progressive

20. 20 Sensor Chip Formats Number of Pixels from 500x500 to 5000x5000 Pixel size from 4  m x 4  m to 16  m x 16  m

21. 21 Line scan camera

22. 22 Image formation - the lens

23. 23 The lens A lens focuses a bundle of rays to one point Parallel rays pass through a focal point at a distance F beyond the plane of the lens. F is the focal length O is the optical center F and O span the optical axis

24. 24 Thin Lens & Gaussian Formula

25. 25 Focus and depth-of-field Depth-of-field (DK: dybteskarphed) Distance range in which the blur does not exceed a certain value

26. 26 Aperture (DK: blænde) More aperture => better depth-of-field Downside: less light enters => decrease exposure time => risk of blur due to motion Depth-of-field

27. 27 Field-of-view Field-of-view depends on size of chip and focal length ”Fisheye” lens => small focal length and large field-of-view

28. 28 Lighting in machine vision

29. 29 Levels of Natural Light [Burke]

30. 30

31. 31

32. 32

33. 33

34. 34 Lighting ”In machine vision lighting is more than 50%” Use controlled lighting! Avoid direct/indirect sunlight –Build a housing covering the field-of-view of the camera Avoid highlights

35. 35 Illumination Setups Directed illumination Rear illumination Light field illumination Diffuse illumination

36. 36 Backlighting

37. 37

38. 38

39. 39

40. 40

41. 41

42. 42 Viewpoint invariant High reflectance in illumination direction Spherical Marker

43. 43 Infrared Illumination Visually Opaque IR pass filter Fast segmentation by adaptive threshold Robust

44. 44 What to remember

45. 45 Exercises (1/3) Given a 512 x 512 x 8bit image. How many different images can be made? Given a 512 x 512 x 8bit image. How is the memory size reduced when you: –Decrease the grayscale resolution repeatedly by 2 –Decrease the x-size and y-size of the image repeatedly by 2 What do you want to learn in this class?

46. 46 Exercises (2/3) Describe the following concepts and provide examples of their usages: –Classification, Quality control, Pose estimation, Bin- picking What is a CCD-chip and how does it operate? What is Depth-of-field (DK:dybteskarphed)? Pros and cons of back-lighting?

47. 47 Exercises (3/3) Show that the following is true for a thin lens: Mick is 2m tall and standing 5m from a camera. The camera’s focal length is 5mm. –At which distance from the lens is Mick in focus? How tall (in mm) will Mick be on the CCD-chip? How tall (in pixels) will Mick be on the CCD-chip? –The camera has a 1/2” CCD chip –The camera image has a size of: 480x640 pixels What is field-of-view of the camera?

48. 48 Working with images…. Image manipulation –Simple operations, e.g., scale image Image processing –Improve the image, e.g., remove noise Image analysis –Analyze the image, e.g., find the person in the image Machine vision –Industry, e.g., Quality control, Robot control Computer vision –Everything: multiple cameras, video-processing, etc.

49. 49 Image file types image.jpg, image.tif, image.gif, image.png, image.ppm, …. Raw: –No data is lost –Header + data (234 235 32 21…) –For example: image.pgm –The file can be viewed Lossless compression: –No data is lost, but the file cannot be viewed –For example: image.gif Lossy compression: –Better compression –Some data is lost (optimized from the HVS’ point of view) –The file cannot be viewed –For example: image.jpg

50. 50 Image file types Normally you don’t care about the file type –The application will take care of it for you: –For example: rotate Application –image.x => raw –Rotate the raw image –Rotated raw => rotated_image.x But to write your own programs from scratch the images need to be in the raw format (without a header).

51. 51 Video Standards RS-170 –NTSC CCIR –PAL, SECAM USB, Firewire Composite S-video RGB

52. 52 Applications

53. 53 Digital Image Why digital ? Before 1920: Image transmission from USA to Europe: more than a week: by ship! Early 1920s: Bartline cable picture transmission system: Transmission in three hours! Transmission via telegraph/wire, radio signals for newspapers

54. 54 Small Progress Small progress in digital imaging until 1964 Jet Propulsion Lab (JPL) in Pasadena, CA –Transmission and correction of lunar images from Ranger 7. Not so good quality so the images had to be processed before they could be viewed Since then many applications…

55. 55 Examples: Image Correction Needed when image data is erroneous: –Bad transmission –Bits are missing: Salt & Pepper Noise

56. 56 Image Deblurring: Motion Blur Can be used when a camera or object is moved during exposure

57. 57 Deblurring Can be used when the camera was not focused properly!!

58. 58 Image manipulation Image improvement, e.g. too dark image Rotate + scale

59. 59 Medical Image Processing Image Processing is widely used E.g. Analysis of microscopic images

60. 60 Medical Image Processing MR/CT Imaging of a human body Use for Brain Surgery

61. 61 Biometrics Recognizing/verifying the identity of a person by analyzing one or more characteristics of the human body Characteristics: –Fingerprint, eye (retina, iris), ear, face, heat profile, shape (3D face, hand), motion (gait, writing), … Applications: –Verifying: Access control (bio-passports) –Recognizing: Surveillance: 9/11

62. 62 Chroma keying

63. 63 Analysis of Sport Motions Here: Analysis of motion of Sarah Hughes 3D Tracking of body parts Motion interpretation Action recognition

64. 64 Motion Capture Andy Serkis Special effects –Advertising –Movies

65. 65 Motion Capture

66. 66 Fundamental Steps in Computer Vision Knowledge base Problem domain Image acquisition Preprocessing Segmentation Representation and description Recognition and Interpretation Result Point 1: 22,33 Point 2: 24, 39 ….. Actor sitting

67. 67 Shrinking the pinhole [from Hecht 1987]

68. 68 Shrinking the pinhole even more [from Hecht 1987] Diffraction causes blur

69. 69 Basic Optics Reflection Refraction & Dispersion Diffraction

70. 70 Minimum Object Distance