National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Using Image Data in Your Research Kenton McHenry, Ph.D. Research Scientist

Image and Spatial Data Analysis Group

Research & Development Cyberinfrastructure: Software development for the sciences (and industry) Computer Vision: Information from images High Performance Computing: Software that scales with regards to computation and data

Image and Spatial Data Analysis Group Content Based Retrieval Search in digitized collections Document segmentation Authorship 3D models Automatic Image Annotation Assign keywords as metadata Tracking 3D Reconstruction Image Stitching

Image and Spatial Data Analysis Group Digital Preservation Access to data content independent of format Access to software functionality independent of distribution Information loss evaluation Document similarity Environmental Modeling Workflows Heterogeneous data sources Data Exploration Data mining eScience

Goals for Today A high level understanding of what Computer Vision is and how YOU might use it. A sense of what is currently possible A sense of how these things break A sense of what might be possible A sense of what is pure science fiction! The looming opportunity in “Big Data” A little bit of hands on experience

Computer Vision Books: D. Forsyth, J. Ponce, “Computer Vision: A Modern Approach”, Pearson, R. Szeliski, “Computer Vision: Algorithms and Applications”, CS 543: Computer Vision (UIUC) Derek Hoiem, Ph.D.

Computer Vision [Hoiem, 2012]

Computer Vision Make a computer understand images and video What kind of scene? Are there cars? Where are the cars? Is it day or night? What is the ground made of? How far is the building? [Hoiem, 2012]

Raster Images [Hoiem, 2012]

Image Creation Light emitted Sensor Lens Fraction of light reflects into camera [Hoiem, 2012]

Image Creation Light(s) Position Strength Geometry Color Surface(s) Orientation Color Material Nearby surfaces Sensor Lens Aperture Exposure Resolution Light emitted Sensor Light reflected to camera [Hoiem, 2012]

Surfaces: Reflected Light incoming light specular reflection incoming light diffuse reflection absorption incoming light [Hoiem, 2012]

Surface: Reflected Light

Surfaces: Orientation 1 2 I x =  x LN x [Hoiem, 2012]

Surfaces light sourcetransparency light source refraction [Hoiem, 2012]

Surfaces λ1λ1 light source λ2λ2 fluorescence

Surfaces t=1 light source t>1 phosphorescence [Hoiem, 2012]

Surfaces λ light source subsurface scattering [Hoiem, 2012]

Light Human Luminance Sensitivity Function [Hoiem, 2012]

Light [Hoiem, 2012]

Light

[GIMP Demo]

Sensors Long (red), Medium (green), and Short (blue) cones, plus intensity rods [Hoiem, 2012]

Sensors [Hoiem, 2012]

Sensors R G B [Hoiem, 2012]

Sensors: Perspective Projecting a 3D world onto a 2D plane Parallel lines disappear at vanishing points Sizes appear smaller further away

Surface Interactions! [Hoiem, 2012]

Surface Interactions [Hoiem, 2012]

Surface Interactions [Hoiem, 2012]

Surfaces: Interactions

Surface Interactions [Hoiem, 2012]

Raster Images [Hoiem, 2012] image(234, 452) = 0.58

Individual Pixels [Hoiem, 2012]

Neighborhoods of Pixels For nearby surface points most factors do not change much Local differences in brightness [Hoiem, 2012]

Neighborhoods of Pixels [Hoiem, 2012]

Neighborhoods of Pixels [Hoiem, 2012]

Neighborhoods of Pixels [Hoiem, 2012]

Changes in Intensity Changes in albedo Changes in surface normal Changes in distance [Hoiem, 2012]

Computer Vision Make a computer understand images and video Lots of variables are involved in the creation of an image/frame Variables are not independent and interact The problem is underconstraned i.e. multiple scenes can result in the same image

Optical Illusions

Vision is Really Hard! Vision is an amazing feat of natural intelligence More human brain devoted to vision than anything else [Hoiem, 2012]

State of the Art From 1960’s to present…

Barcodes Optical machine readable representation of data 1950’s

Optical Character Recognition (OCR) Digit recognition, AT&T labs Technology to convert scanned documents to ASCII text If you have a scanner, it probably came with OCR software License plate readers [Hoiem, 2012]

Biometrics Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely [Hoiem, 2012]

Face detection Many new digital cameras now detect faces Canon, Sony, Fuji, … [Hoiem, 2012]

Medical imaging 3D imaging, MRI, CT [Hoiem, 2012],

The Matrix movies, ESC Entertainment, XYZRGB, NRC 3D Reconstruction

Pirates of the Carribean, Industrial Light and Magic Motion capture [Hoiem, 2012]

Image Stitching NASA'S Mars Exploration Rover Spirit NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of [Hoiem, 2012]

Industry Vision-guided robots position nut runners on wheels [Hoiem, 2012]

Sports [Hoiem, 2012]

Object Recognition Point & FindPoint & Find, Nokia, Google GogglesNokiaGoogle Goggles [Hoiem, 2012] LaneHawk by EvolutionRobotics

Human Computer Interaction Object Recognition: 3D Reconstruction: Robot: [Hoiem, 2012]

Driving Oct 9, "Google Cars Drive Themselves, in Traffic". The New York Times. John Markoff"Google Cars Drive Themselves, in Traffic"The New York Times June 24, "Nevada state law paves the way for driverless cars". Financial Post. Christine Dobby"Nevada state law paves the way for driverless cars"Financial Post Aug 9, 2011, "Human error blamed after Google's driverless car sparks five-vehicle crash". The Star (Toronto)"Human error blamed after Google's driverless car sparks five-vehicle crash" [Hoiem, 2012]

State of the Art Remember vision is hard! Most vision applications are “quirky”.

Image and Spatial Data Analysis Group Questions?