2. Vision, Video and Virtual Reality Sensors Lecture 4 Sensors CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html

3. Vision, Video and Virtual Reality Acknowledgements The slides in this lecture were adopted and modified from lectures by Professor Allen Hanson University of Massachusetts at Amherst

4. Vision, Video and Virtual Reality Sensors n Static monocular reflectance data (monochromic or color) l Films l Video cameras (with tapes) l Digital cameras (with memory) n Motion sequences (camcorders) n Stereo (2 cameras) n Range data (Range finder) n Non-visual sensory data l infrared (IR) l ultraviolet (UV) l microwaves n Many more

5. Vision, Video and Virtual Reality The Electromagnetic Spectrum Visible Spectrum 700 nm 400 nm C = f  f

6. Vision, Video and Virtual Reality The Human Eye

7. Vision, Video and Virtual Reality The Eye n The Retina: l rods (low-level light, night vision) l cones (color-vision) l synapses l optic nerve fibers n Sensing and low-level processing layer l 125 millions rods and cones feed into 1 million nerve fibers l Cell arrangement that respond to horizontal and vertical lines Retina Rods Cones

8. Vision, Video and Virtual Reality Film, Video, Digital Cameras n Black and White (Reflectance data only) n Color (Reflectance data in three bands - red, green, blue)

9. Vision, Video and Virtual Reality Color Images BlueGreenRed ‘Dimensions’ of an Image Spatial (x,y) Depth (no. of components) Number of bits/channel Temporal (t) Pixel Spatial Resolution Spectra Resolution Radiometric Resolution Temporal Resolution

10. Vision, Video and Virtual Reality Across the EM Spectrum Crab Nebula

11. Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

12. Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

13. Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 10 19 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

14. Vision, Video and Virtual Reality Across the EM Spectrum n Medical X-Rays

15. Vision, Video and Virtual Reality Across the EM Spectrum n Chandra X-Ray Satellite

16. Vision, Video and Virtual Reality Across the EM Spectrum n From X-Ray images to 3D Models: CT Scans

17. Vision, Video and Virtual Reality Across the EM Spectrum n Flower Patterns in Ultraviolet Dandelion - UV Potentilla

18. Vision, Video and Virtual Reality Across the EM Spectrum n Messier 101 in Ultraviolet

19. Vision, Video and Virtual Reality Across the EM Spectrum n Traditional images

20. Vision, Video and Virtual Reality Across the EM Spectrum n Non-traditional Use of Visible Light: Range

21. Vision, Video and Virtual Reality Across the EM Spectrum n Scanning Laser Rangefinder

22. Vision, Video and Virtual Reality Across the EM Spectrum n IR: Near, Medium, Far (~heat)

23. Vision, Video and Virtual Reality Across the EM Spectrum n IR: Near, Medium, Far (~heat)

24. Vision, Video and Virtual Reality Across the EM Spectrum n IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

25. Vision, Video and Virtual Reality Across the EM Spectrum n (Un)Common uses of Microwaves CD Movie Exploding Water Movie

26. Vision, Video and Virtual Reality Across the EM Spectrum n Microwave Imaging: Synthetic Aperture Radar (SAR) San Fernando Valley Tibet: Lhasa River Thailand: Phang Hoei Range Athens, Greece Red: L-band (24cm) Green: C-band (6 cm) Blue:C/L

27. Vision, Video and Virtual Reality Across the EM Spectrum n Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR (elevation)

28. Vision, Video and Virtual Reality Across the EM Spectrum n Low Altitude IFSAR IFSAR elevation, automatic, in minutes Elevation from aerial stereo, manually, several days

29. Vision, Video and Virtual Reality Across the EM Spectrum Radio Waves (images of cosmos from radio telescopes)

30. Vision, Video and Virtual Reality Stereo Geometry n Single Camera (no stereo)

31. Vision, Video and Virtual Reality Stereo Geometry P(X,Y,Z) f = focal length Optical Center p r (x,y) Film plane p l (x,y) Optical Center f = focal length Film plane LEFT CAMERARIGHT CAMERA B = Baseline

32. Vision, Video and Virtual Reality Stereo Geometry LEFT IMAGE RIGHT IMAGE Disparity = x r - x l P P r (x r,y r )P l (x l,y l ) ≈ depth

33. Vision, Video and Virtual Reality Stereo Images n A Short Digression Stereoscopes

34. Vision, Video and Virtual Reality Stereo Images Darjeeling Suspension Bridge

35. Vision, Video and Virtual Reality Picture of you?

36. Vision, Video and Virtual Reality Stereo n Stereograms

37. Vision, Video and Virtual Reality Stereo X-Ray

38. Vision, Video and Virtual Reality Range Sensors n Light Striping David B. Cox, Robyn Owens and Peter Hartmann Department of Biochemistry University of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/

39. Vision, Video and Virtual Reality Mosaics n A mosaic is created from several images

40. Vision, Video and Virtual Reality Mosaics n Stabilized Video

41. Vision, Video and Virtual Reality Mosaics n Depth from a Video Sequence (single camera) P(X,Y,Z) Height H from Laser Profiler GPS

42. Vision, Video and Virtual Reality Mosaics n Brazilian forest…..made at UMass CVL

43. Vision, Video and Virtual Reality Why is Vision Difficult? n Natural Variation in Object Classes: l Color, texture, size, shape, parts, and relations n Variations in the Imaging Process l Lighting (highlights, shadows, brightness, contrast) l Projective distortion, point of view, occlusion l Noise, sensor and optical characteristics n Massive Amounts of Data l 1 minute of 1024x768 color video = 4.2 gigabytes (Uncompressed)

44. Vision, Video and Virtual Reality The Need for Knowledge Knowledge Function Context Shape Specific Objects Generic Objects Structure Size Shape Motion Purpose Variation

45. Vision, Video and Virtual Reality The Figure Revealed

46. Vision, Video and Virtual Reality The Effect of Context

47. Vision, Video and Virtual Reality The Effect of Context - 2

48. Vision, Video and Virtual Reality Context, cont. n ….a collection of objects:

49. Vision, Video and Virtual Reality Context n The objects as hats:

50. Vision, Video and Virtual Reality n And as something else….. n ‘To interpret something is to give it meaning in context.’ Context

51. Vision, Video and Virtual Reality Vision System Components n …..at the low (image) level, we need l Ways of generating initial descriptions of the image data l Method for extracting features of these descriptions l Ways of representing these descriptions and features l Usually, cannot initially make use of general world knowledge IMAGE (numbers) DESCRIPTION (symbols)

52. Vision, Video and Virtual Reality n ….at the intermediate level, we need l Symbolic representations of the initial descriptions l Ways of generating more abstract descriptions from the initial ones (grouping) l Ways of accessing relevant portions of the knowledge base l Ways of controlling the processing n Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data- directed) IMAGE IINTERMEDIATE DESCRIPTIONS KNOWLEDGE Vision System Components

53. Vision, Video and Virtual Reality Vision System Components n ….at the high (interpretation) level, we need l Ways of representing world knowledge n Objects n Object parts n Expected scenarios (relations) n Specializations l Mechanisms for Interferencing n Beliefs n Partial matches l Control Information l Representations of n Partial interpretations n Competing interpretations n Relationship to the image descriptions

54. Vision, Video and Virtual Reality Next Anyone who isn't confused really doesn't understand the situation. --Edward R. Murrow Next: Image Formation Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5 Questions: 2.1. 2.2, 2.3, 2.5 Exercises: 2.1, 2.3, 2.4