Computers and Photographs 1) Image Processing 2) Computer Vision Henry Schneiderman

Outline Digital Cameras Emerging Technology Research in Image Processing and Computer Vision Automatically Finding Faces and Cars in Photographs

Digital Cameras = Convenience Easy to capture photos Easy to store and organize photos Easy to duplicate photos Easy to edit photos

Digital Camera Usage Lyra research report, 1999 Exposures in billions

State of the Art: Digital Cameras Film is currently better in resolution and color. –Professional photographers Digital for low quality newspaper adds Film for portrait photos Computer storage limitations: 1 high resolution digital image = 25 MegaBytes Printing –home printers not comparable to commercial printers

Future of Digital Cameras Improved resolution and color “Smart” cameras More programmable features –Auto-focus on object of interest –“Everything in focus” photo –Capture photo when event X occurs

Photographs: Migration to Digital Format Others means of digitizing imagery –Scanners (photo and film) –Frame-grabber for video

Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games,...

Optical Character Recognition (OCR) First patent in OCR in 19 th century First applications in post-office and banks State of the art not perfect. Examples of errors:

Handwriting Recognition Works if constraints on writer, e.g. palm pilot

Other document processing Not just for text... Examples: –Engineering document to CAD file –Maps to GIS format –Music score to MIDI representation

Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

Biometrics Technology for identification –Finger/palm print –Iris –Face

Fingerprints Minutae – spits and merges of ridges

Face Identification Not quite reliable yet. –Performance degrades rapidly with uncontrolled lighting, facial expression, and size of database Several companies exist: –Visionics (Rockfeller Univesity spin-off) –EyeMatic (USC spin-off) –Miros (MIT spin-off) –Banque-Tec Intl (Australia) –C-VIS Computer Vision (Germany) –LAU Technologies Commercial systems installed in London and Brazil to catch criminals

Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

Management of images on computers Compression – reducing storage size needed for images Watermarking – Protecting copyright Microsoft, Bell Labs, NEC, etc. Visible watermark

Photo-manipulation Adobe Photoshop, Corel PhotoPaint, Pixami, PhotoIQ, etc. –More automatic features

Searching Digital Image Collections Large collections of images exist –Corbis 67 million images –Getty 70 million stock photography images –AP collects 1000s of digitized images per day Search methods are inadequate –Rely on captions and colors IBM’s Query by Image Content (QBIC) system

Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

Inspection for Manufacturing Occum – inspection of printed circuit boards ($100M / year) Cognex – Do-it-yourself toolkits for inspection (400 employees)

Automatic Target Recognition (ATR) Finding mines, tanks, etc. Billion dollar a year industry –Martin-Lockheed, TSR, Northrup-Grumman, other aerospace contractors. Various types of imagery: –Synthetic Aperture Radar (SAR), Sonar, hyper- spectral imagery (more than 3 colors)

Aerial Photo Interpretation / Automated Cartography Classification of land-use: forest, vegetation, water Identification of man-made objects: buildings, roads, etc

Better Security Cameras Cameras that are responsive to the environment –Track and zoom on moving objects –Automatic adjustment of contrast

Human-Computer Interaction Computer games that involve interaction with user Intelligent teleconferencing

Medical imagery Medical image libraries for study and diagnosis Image overlay to guide surgeons

History 1980’s ~100 companies – manufacturing applications mostly Early 1990’s less than 10 companies Late 1990’s ~100 companies – face recognition, intelligent teleconferencing, inspection, digital libraries

Computer Vision and Image Processing Research

Image processing Computer vision chair, face, shape, etc. “Symbolic” descriptionimage

1. Image Processing: Filtering

2. Image Processing: Compression Lossless – No loss in quality, gif, tiff Lossy – Original image cannot be reconstructed, jpeg

3. Image Processing: Watermarking Information hiding –Protecting Copyright

4. Image Processing: Transformation Transforming image can make it easier to analyze Wavelet transform of image

Decomposition in Resolution/Frequency fine coarseintermediate

Wavelet Decomposition Vertical subbands (LH)

Wavelet Decomposition Horizontal subbands (HL)

1. Computer Vision: 3D Shape Reconstruction Use images to build 3D model of object or site 3D site model built from laser range scans collected by CMU autonomous helicopter

2. Computer Vision: To guide Motion Visually guided locomotion –robotic vehicles Visually guided manipulation –Hand-eye coordination CMU NavLab II

3. Computer Vision: Recognition and Classification

Challenges in Object Recognition

Challenges in Object Detection Intra-class variation

Lighting variation

Geometric variation

Simpler Problem: Classification Fixed size input Fixed object size, orientation, and alignment “Object is present” (at fixed size and alignment) “Object is NOT present” (at fixed size and alignment) Decision

1) Apply Local Operators f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723

2) Look-Up Probabilities f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723 P 1 ( #5710, 0, 0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214, 0, 1 | obj) = 0.57 P 1 ( #3214, 0, 1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = 0.19

Probabilities Estimated Off-Line f 1 (0, 0) = #567H 1 (#567, 0, 0) = H 1 (567, 0, 0) + 1 f k (n, m) = #350H k (#350, 0, 0) = H k (#350, 0, 0) + 1 P 1 (#567, 0, 0) =  H 1 (#i, 0, 0) H 1 (#567, 0, 0) P k (#350, 0, 0) =  H k (#i, 0, 0) H k (#350, 0, 0)

3) Make Decision P 1 ( #5710, 0, 0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214, 0, 1 | obj) = 0.57 P 1 ( #3214, 0, 1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = * 0.57 *... * * 0.48 *... * 0.19 >

Overall Algorithm f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723 P 1 ( #5710,0,0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214,0,1 | obj) = 0.57 P 1 ( #3214,0,1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = * 0.57 *... * * 0.48 * … * 0.19

Detection: Apply Classifier Exhaustively Search in position Search in scale

View-based Classifiers Face Classifier #1 Face Classifier #2 Face Classifier #3

2 classifiers trained for faces. 8 classifiers trained for cars.

Training Classifiers Cars: images per viewpoint Faces: 2,000 images per viewpoint ~1,000 synthetic variations of each original image –background scenery, orientation, position, frequency 2000 non-object images –Samples selected by bootstrapping Minimization of classification error on training set –AdaBoost algorithm (Freund & Shapire ‘97, Shapire & Singer ‘99) Iterative method Determines weights for samples

Applications of Face Detection Automatic red-eye removal from photographs Automatic color balancing in photo- finishing Intelligent teleconferencing Component in face identification system

Difficulty Increases with Complexity of Object 2D vs. 3D Specific objects – e.g. my coffee mug A category of objects – e.g. all coffee mugs Amount of intra-category variation –Rigid or semi-rigid structure, e.g. face –Articulated objects, e.g. human body –Functionally defined objects, e.g. chairs

Summary: Image Processing & Computer Vision Not as mature as speech recognition –Technology not as reliable –Fewer companies, fewer products Success on limited problems, e.g., documents More applicable to fault tolerant problems Technology will grow –Emergence of digital camera –Improved methods