Vision, Video and Virtual Reality Sensors Lecture 4 Sensors CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A Capstone2004/Capstone_Sequence2004.html

Vision, Video and Virtual Reality Acknowledgements The slides in this lecture were adopted and modified from lectures by Professor Allen Hanson University of Massachusetts at Amherst

Vision, Video and Virtual Reality Sensors n Static monocular reflectance data (monochromic or color) l Films l Video cameras (with tapes) l Digital cameras (with memory) n Motion sequences (camcorders) n Stereo (2 cameras) n Range data (Range finder) n Non-visual sensory data l infrared (IR) l ultraviolet (UV) l microwaves n Many more

Vision, Video and Virtual Reality The Electromagnetic Spectrum Visible Spectrum 700 nm 400 nm C = f  f

Vision, Video and Virtual Reality The Human Eye

Vision, Video and Virtual Reality The Eye n The Retina: l rods (low-level light, night vision) l cones (color-vision) l synapses l optic nerve fibers n Sensing and low-level processing layer l 125 millions rods and cones feed into 1 million nerve fibers l Cell arrangement that respond to horizontal and vertical lines Retina Rods Cones

Vision, Video and Virtual Reality Film, Video, Digital Cameras n Black and White (Reflectance data only) n Color (Reflectance data in three bands - red, green, blue)

Vision, Video and Virtual Reality Color Images BlueGreenRed ‘Dimensions’ of an Image Spatial (x,y) Depth (no. of components) Number of bits/channel Temporal (t) Pixel Spatial Resolution Spectra Resolution Radiometric Resolution Temporal Resolution

Vision, Video and Virtual Reality Across the EM Spectrum Crab Nebula

Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Vision, Video and Virtual Reality Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Vision, Video and Virtual Reality Across the EM Spectrum n Medical X-Rays

Vision, Video and Virtual Reality Across the EM Spectrum n Chandra X-Ray Satellite

Vision, Video and Virtual Reality Across the EM Spectrum n From X-Ray images to 3D Models: CT Scans

Vision, Video and Virtual Reality Across the EM Spectrum n Flower Patterns in Ultraviolet Dandelion - UV Potentilla

Vision, Video and Virtual Reality Across the EM Spectrum n Messier 101 in Ultraviolet

Vision, Video and Virtual Reality Across the EM Spectrum n Traditional images

Vision, Video and Virtual Reality Across the EM Spectrum n Non-traditional Use of Visible Light: Range

Vision, Video and Virtual Reality Across the EM Spectrum n Scanning Laser Rangefinder

Vision, Video and Virtual Reality Across the EM Spectrum n IR: Near, Medium, Far (~heat)

Vision, Video and Virtual Reality Across the EM Spectrum n IR: Near, Medium, Far (~heat)

Vision, Video and Virtual Reality Across the EM Spectrum n IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

Vision, Video and Virtual Reality Across the EM Spectrum n (Un)Common uses of Microwaves CD Movie Exploding Water Movie

Vision, Video and Virtual Reality Across the EM Spectrum n Microwave Imaging: Synthetic Aperture Radar (SAR) San Fernando Valley Tibet: Lhasa River Thailand: Phang Hoei Range Athens, Greece Red: L-band (24cm) Green: C-band (6 cm) Blue:C/L

Vision, Video and Virtual Reality Across the EM Spectrum n Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR (elevation)

Vision, Video and Virtual Reality Across the EM Spectrum n Low Altitude IFSAR IFSAR elevation, automatic, in minutes Elevation from aerial stereo, manually, several days

Vision, Video and Virtual Reality Across the EM Spectrum Radio Waves (images of cosmos from radio telescopes)

Vision, Video and Virtual Reality Stereo Geometry n Single Camera (no stereo)

Vision, Video and Virtual Reality Stereo Geometry P(X,Y,Z) f = focal length Optical Center p r (x,y) Film plane p l (x,y) Optical Center f = focal length Film plane LEFT CAMERARIGHT CAMERA B = Baseline

Vision, Video and Virtual Reality Stereo Geometry LEFT IMAGE RIGHT IMAGE Disparity = x r - x l P P r (x r,y r )P l (x l,y l ) ≈ depth

Vision, Video and Virtual Reality Stereo Images n A Short Digression Stereoscopes

Vision, Video and Virtual Reality Stereo Images Darjeeling Suspension Bridge

Vision, Video and Virtual Reality Picture of you?

Vision, Video and Virtual Reality Stereo n Stereograms

Vision, Video and Virtual Reality Stereo X-Ray

Vision, Video and Virtual Reality Range Sensors n Light Striping David B. Cox, Robyn Owens and Peter Hartmann Department of Biochemistry University of Western Australia

Vision, Video and Virtual Reality Mosaics n A mosaic is created from several images

Vision, Video and Virtual Reality Mosaics n Stabilized Video

Vision, Video and Virtual Reality Mosaics n Depth from a Video Sequence (single camera) P(X,Y,Z) Height H from Laser Profiler GPS

Vision, Video and Virtual Reality Mosaics n Brazilian forest…..made at UMass CVL

Vision, Video and Virtual Reality Why is Vision Difficult? n Natural Variation in Object Classes: l Color, texture, size, shape, parts, and relations n Variations in the Imaging Process l Lighting (highlights, shadows, brightness, contrast) l Projective distortion, point of view, occlusion l Noise, sensor and optical characteristics n Massive Amounts of Data l 1 minute of 1024x768 color video = 4.2 gigabytes (Uncompressed)

Vision, Video and Virtual Reality The Need for Knowledge Knowledge Function Context Shape Specific Objects Generic Objects Structure Size Shape Motion Purpose Variation

Vision, Video and Virtual Reality The Figure Revealed

Vision, Video and Virtual Reality The Effect of Context

Vision, Video and Virtual Reality The Effect of Context - 2

Vision, Video and Virtual Reality Context, cont. n ….a collection of objects:

Vision, Video and Virtual Reality Context n The objects as hats:

Vision, Video and Virtual Reality n And as something else….. n ‘To interpret something is to give it meaning in context.’ Context

Vision, Video and Virtual Reality Vision System Components n …..at the low (image) level, we need l Ways of generating initial descriptions of the image data l Method for extracting features of these descriptions l Ways of representing these descriptions and features l Usually, cannot initially make use of general world knowledge IMAGE (numbers) DESCRIPTION (symbols)

Vision, Video and Virtual Reality n ….at the intermediate level, we need l Symbolic representations of the initial descriptions l Ways of generating more abstract descriptions from the initial ones (grouping) l Ways of accessing relevant portions of the knowledge base l Ways of controlling the processing n Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data- directed) IMAGE IINTERMEDIATE DESCRIPTIONS KNOWLEDGE Vision System Components

Vision, Video and Virtual Reality Vision System Components n ….at the high (interpretation) level, we need l Ways of representing world knowledge n Objects n Object parts n Expected scenarios (relations) n Specializations l Mechanisms for Interferencing n Beliefs n Partial matches l Control Information l Representations of n Partial interpretations n Competing interpretations n Relationship to the image descriptions

Vision, Video and Virtual Reality Next Anyone who isn't confused really doesn't understand the situation. --Edward R. Murrow Next: Image Formation Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5 Questions: , 2.3, 2.5 Exercises: 2.1, 2.3, 2.4