2. Introduction to Computer Vision Sensors Instructor: Zhigang Zhu CSc80000 Section 2 Spring 2005 3D Computer Vision and Video Computing Lecture 2 –Part 1: Sensors http://www-cs.engr.ccny.cuny.edu/~zhu/GC-Spring2005/CSc80000-2-VisionCourse.html

3. Introduction to Computer Vision Acknowledgements The slides in this lecture were adopted and modified from lectures by Professor Allen Hanson University of Massachusetts at Amherst

4. Introduction to Computer Vision Sensors n Static monocular reflectance data (monochromic or color) l Films l Video cameras (with tapes) l Digital cameras (with memory) n Motion sequences (camcorders) n Stereo (2 cameras) n Range data (Range finder) n Non-visual sensory data l infrared (IR) l ultraviolet (UV) l microwaves n Many more

5. Introduction to Computer Vision The Electromagnetic Spectrum Visible Spectrum 700 nm 400 nm C = f  f

6. Introduction to Computer Vision The Human Eye

7. Introduction to Computer Vision The Eye n The Retina: l rods (low-level light, night vision) l cones (color-vision) l synapses l optic nerve fibers n Sensing and low-level processing layer l 125 millions rods and cones feed into 1 million nerve fibers l Cell arrangement that respond to horizontal and vertical lines Retina Rods Cones

8. Introduction to Computer Vision Film, Video, Digital Cameras n Black and White (Reflectance data only) n Color (Reflectance data in three bands - red, green, blue)

9. Introduction to Computer Vision Color Images BlueGreenRed ‘Dimensions’ of an Image Spatial (x,y) Depth (no. of components) Number of bits/channel Temporal (t) Pixel Spatial Resolution Spectra Resolution Radiometric Resolution Temporal Resolution

10. Introduction to Computer Vision Across the EM Spectrum Crab Nebula

11. Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

12. Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

13. Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

14. Introduction to Computer Vision Across the EM Spectrum n Medical X-Rays

15. Introduction to Computer Vision Across the EM Spectrum n Chandra X-Ray Satellite

16. Introduction to Computer Vision Across the EM Spectrum n From X-Ray images to 3D Models: CT Scans

17. Introduction to Computer Vision Across the EM Spectrum n Flower Patterns in Ultraviolet Dandelion - UV Potentilla

18. Introduction to Computer Vision Across the EM Spectrum n Messier 101 in Ultraviolet

19. Introduction to Computer Vision Across the EM Spectrum n Traditional images

20. Introduction to Computer Vision Across the EM Spectrum n Non-traditional Use of Visible Light: Range

21. Introduction to Computer Vision Across the EM Spectrum n Scanning Laser Rangefinder

22. Introduction to Computer Vision Across the EM Spectrum n IR: Near, Medium, Far (~heat)

23. Introduction to Computer Vision Across the EM Spectrum n IR: Near, Medium, Far (~heat)

24. Introduction to Computer Vision Across the EM Spectrum n IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

25. Introduction to Computer Vision Across the EM Spectrum n (Un)Common uses of Microwaves CD Movie Exploding Water Movie

26. Introduction to Computer Vision Across the EM Spectrum n Microwave Imaging: Synthetic Aperture Radar (SAR) San Fernando Valley Tibet: Lhasa River Thailand: Phang Hoei Range Athens, Greece Red: L-band (24cm) Green: C-band (6 cm) Blue:C/L

27. Introduction to Computer Vision Across the EM Spectrum n Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR (elevation)

28. Introduction to Computer Vision Across the EM Spectrum n Low Altitude IFSAR IFSAR elevation, automatic, in minutes Elevation from aerial stereo, manually, several days

29. Introduction to Computer Vision Across the EM Spectrum Radio Waves (images of cosmos from radio telescopes)

30. Introduction to Computer Vision Stereo Geometry n Single Camera (no stereo)

31. Introduction to Computer Vision Stereo Geometry P(X,Y,Z) f = focal length Optical Center p r (x,y) Film plane p l (x,y) Optical Center f = focal length Film plane LEFT CAMERARIGHT CAMERA B = Baseline

32. Introduction to Computer Vision Stereo Geometry LEFT IMAGE RIGHT IMAGE Disparity = x r - x l P P r (x r,y r )P l (x l,y l ) ≈ depth

33. Introduction to Computer Vision Stereo Images n A Short Digression Stereoscopes

34. Introduction to Computer Vision Stereo Images Darjeeling Suspension Bridge

35. Introduction to Computer Vision Picture of you?

36. Introduction to Computer Vision Stereo n Stereograms

37. Introduction to Computer Vision Stereo X-Ray

38. Introduction to Computer Vision Range Sensors n Light Striping David B. Cox, Robyn Owens and Peter Hartmann Department of Biochemistry University of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/

39. Introduction to Computer Vision Mosaics n A mosaic is created from several images

40. Introduction to Computer Vision Mosaics n Stabilized Video

41. Introduction to Computer Vision Mosaics n Depth from a Video Sequence (single camera) P(X,Y,Z) Height H from Laser Profiler GPS

42. Introduction to Computer Vision Mosaics n Brazilian forest…..made at UMass CVL

43. Introduction to Computer Vision Why is Vision Difficult? n Natural Variation in Object Classes: l Color, texture, size, shape, parts, and relations n Variations in the Imaging Process l Lighting (highlights, shadows, brightness, contrast) l Projective distortion, point of view, occlusion l Noise, sensor and optical characteristics n Massive Amounts of Data l 1 minute of 1024x768 color video = 4.2 gigabytes (Uncompressed)

44. Introduction to Computer Vision The Need for Knowledge Knowledge Function Context Shape Specific Objects Generic Objects Structure Size Shape Motion Purpose Variation

45. Introduction to Computer Vision The Figure Revealed

46. Introduction to Computer Vision The Effect of Context

47. Introduction to Computer Vision The Effect of Context - 2

48. Introduction to Computer Vision Context, cont. n ….a collection of objects:

49. Introduction to Computer Vision Context n The objects as hats:

50. Introduction to Computer Vision n And as something else….. n ‘To interpret something is to give it meaning in context.’ Context

51. Introduction to Computer Vision Vision System Components n …..at the low (image) level, we need l Ways of generating initial descriptions of the image data l Method for extracting features of these descriptions l Ways of representing these descriptions and features l Usually, cannot initially make use of general world knowledge IMAGE (numbers) DESCRIPTION (symbols)

52. Introduction to Computer Vision n ….at the intermediate level, we need l Symbolic representations of the initial descriptions l Ways of generating more abstract descriptions from the initial ones (grouping) l Ways of accessing relevant portions of the knowledge base l Ways of controlling the processing n Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data- directed) IMAGE IINTERMEDIATE DESCRIPTIONS KNOWLEDGE Vision System Components

53. Introduction to Computer Vision Vision System Components n ….at the high (interpretation) level, we need l Ways of representing world knowledge n Objects n Object parts n Expected scenarios (relations) n Specializations l Mechanisms for Interferencing n Beliefs n Partial matches l Control Information l Representations of n Partial interpretations n Competing interpretations n Relationship to the image descriptions

54. Introduction to Computer Vision Next Anyone who isn't confused really doesn't understand the situation. --Edward R. Murrow Next: Image Formation Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5 Questions: 2.1. 2.2, 2.3, 2.5 Exercises: 2.1, 2.3, 2.4