Introduction to Computer Vision Sensors Instructor: Zhigang Zhu CSc80000 Section 2 Spring D Computer Vision and Video Computing Lecture 2 –Part 1: Sensors

Introduction to Computer Vision Acknowledgements The slides in this lecture were adopted and modified from lectures by Professor Allen Hanson University of Massachusetts at Amherst

Introduction to Computer Vision Sensors n Static monocular reflectance data (monochromic or color) l Films l Video cameras (with tapes) l Digital cameras (with memory) n Motion sequences (camcorders) n Stereo (2 cameras) n Range data (Range finder) n Non-visual sensory data l infrared (IR) l ultraviolet (UV) l microwaves n Many more

Introduction to Computer Vision The Electromagnetic Spectrum Visible Spectrum 700 nm 400 nm C = f  f

Introduction to Computer Vision The Human Eye

Introduction to Computer Vision The Eye n The Retina: l rods (low-level light, night vision) l cones (color-vision) l synapses l optic nerve fibers n Sensing and low-level processing layer l 125 millions rods and cones feed into 1 million nerve fibers l Cell arrangement that respond to horizontal and vertical lines Retina Rods Cones

Introduction to Computer Vision Film, Video, Digital Cameras n Black and White (Reflectance data only) n Color (Reflectance data in three bands - red, green, blue)

Introduction to Computer Vision Color Images BlueGreenRed ‘Dimensions’ of an Image Spatial (x,y) Depth (no. of components) Number of bits/channel Temporal (t) Pixel Spatial Resolution Spectra Resolution Radiometric Resolution Temporal Resolution

Introduction to Computer Vision Across the EM Spectrum Crab Nebula

Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Introduction to Computer Vision Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

Introduction to Computer Vision Across the EM Spectrum n Medical X-Rays

Introduction to Computer Vision Across the EM Spectrum n Chandra X-Ray Satellite

Introduction to Computer Vision Across the EM Spectrum n From X-Ray images to 3D Models: CT Scans

Introduction to Computer Vision Across the EM Spectrum n Flower Patterns in Ultraviolet Dandelion - UV Potentilla

Introduction to Computer Vision Across the EM Spectrum n Messier 101 in Ultraviolet

Introduction to Computer Vision Across the EM Spectrum n Traditional images

Introduction to Computer Vision Across the EM Spectrum n Non-traditional Use of Visible Light: Range

Introduction to Computer Vision Across the EM Spectrum n Scanning Laser Rangefinder

Introduction to Computer Vision Across the EM Spectrum n IR: Near, Medium, Far (~heat)

Introduction to Computer Vision Across the EM Spectrum n IR: Near, Medium, Far (~heat)

Introduction to Computer Vision Across the EM Spectrum n IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

Introduction to Computer Vision Across the EM Spectrum n (Un)Common uses of Microwaves CD Movie Exploding Water Movie

Introduction to Computer Vision Across the EM Spectrum n Microwave Imaging: Synthetic Aperture Radar (SAR) San Fernando Valley Tibet: Lhasa River Thailand: Phang Hoei Range Athens, Greece Red: L-band (24cm) Green: C-band (6 cm) Blue:C/L

Introduction to Computer Vision Across the EM Spectrum n Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR (elevation)

Introduction to Computer Vision Across the EM Spectrum n Low Altitude IFSAR IFSAR elevation, automatic, in minutes Elevation from aerial stereo, manually, several days

Introduction to Computer Vision Across the EM Spectrum Radio Waves (images of cosmos from radio telescopes)

Introduction to Computer Vision Stereo Geometry n Single Camera (no stereo)

Introduction to Computer Vision Stereo Geometry P(X,Y,Z) f = focal length Optical Center p r (x,y) Film plane p l (x,y) Optical Center f = focal length Film plane LEFT CAMERARIGHT CAMERA B = Baseline

Introduction to Computer Vision Stereo Geometry LEFT IMAGE RIGHT IMAGE Disparity = x r - x l P P r (x r,y r )P l (x l,y l ) ≈ depth

Introduction to Computer Vision Stereo Images n A Short Digression Stereoscopes

Introduction to Computer Vision Stereo Images Darjeeling Suspension Bridge

Introduction to Computer Vision Picture of you?

Introduction to Computer Vision Stereo n Stereograms

Introduction to Computer Vision Stereo X-Ray

Introduction to Computer Vision Range Sensors n Light Striping David B. Cox, Robyn Owens and Peter Hartmann Department of Biochemistry University of Western Australia

Introduction to Computer Vision Mosaics n A mosaic is created from several images

Introduction to Computer Vision Mosaics n Stabilized Video

Introduction to Computer Vision Mosaics n Depth from a Video Sequence (single camera) P(X,Y,Z) Height H from Laser Profiler GPS

Introduction to Computer Vision Mosaics n Brazilian forest…..made at UMass CVL

Introduction to Computer Vision Why is Vision Difficult? n Natural Variation in Object Classes: l Color, texture, size, shape, parts, and relations n Variations in the Imaging Process l Lighting (highlights, shadows, brightness, contrast) l Projective distortion, point of view, occlusion l Noise, sensor and optical characteristics n Massive Amounts of Data l 1 minute of 1024x768 color video = 4.2 gigabytes (Uncompressed)

Introduction to Computer Vision The Need for Knowledge Knowledge Function Context Shape Specific Objects Generic Objects Structure Size Shape Motion Purpose Variation

Introduction to Computer Vision The Figure Revealed

Introduction to Computer Vision The Effect of Context

Introduction to Computer Vision The Effect of Context - 2

Introduction to Computer Vision Context, cont. n ….a collection of objects:

Introduction to Computer Vision Context n The objects as hats:

Introduction to Computer Vision n And as something else….. n ‘To interpret something is to give it meaning in context.’ Context

Introduction to Computer Vision Vision System Components n …..at the low (image) level, we need l Ways of generating initial descriptions of the image data l Method for extracting features of these descriptions l Ways of representing these descriptions and features l Usually, cannot initially make use of general world knowledge IMAGE (numbers) DESCRIPTION (symbols)

Introduction to Computer Vision n ….at the intermediate level, we need l Symbolic representations of the initial descriptions l Ways of generating more abstract descriptions from the initial ones (grouping) l Ways of accessing relevant portions of the knowledge base l Ways of controlling the processing n Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data- directed) IMAGE IINTERMEDIATE DESCRIPTIONS KNOWLEDGE Vision System Components

Introduction to Computer Vision Vision System Components n ….at the high (interpretation) level, we need l Ways of representing world knowledge n Objects n Object parts n Expected scenarios (relations) n Specializations l Mechanisms for Interferencing n Beliefs n Partial matches l Control Information l Representations of n Partial interpretations n Competing interpretations n Relationship to the image descriptions

Introduction to Computer Vision Next Anyone who isn't confused really doesn't understand the situation. --Edward R. Murrow Next: Image Formation Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5 Questions: , 2.3, 2.5 Exercises: 2.1, 2.3, 2.4