1. Computers and Photographs 1) Image Processing 2) Computer Vision Henry Schneiderman

2. Outline Digital Cameras Emerging Technology Research in Image Processing and Computer Vision Automatically Finding Faces and Cars in Photographs

3. Digital Cameras = Convenience Easy to capture photos Easy to store and organize photos Easy to duplicate photos Easy to edit photos

4. Digital Camera Usage Lyra research report, 1999 Exposures in billions

5. State of the Art: Digital Cameras Film is currently better in resolution and color. –Professional photographers Digital for low quality newspaper adds Film for portrait photos Computer storage limitations: 1 high resolution digital image = 25 MegaBytes Printing –home printers not comparable to commercial printers

6. Future of Digital Cameras Improved resolution and color “Smart” cameras More programmable features –Auto-focus on object of interest –“Everything in focus” photo –Capture photo when event X occurs

7. Photographs: Migration to Digital Format Others means of digitizing imagery –Scanners (photo and film) –Frame-grabber for video

8. Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games,...

9. Optical Character Recognition (OCR) First patent in OCR in 19 th century First applications in post-office and banks State of the art not perfect. Examples of errors:

10. Handwriting Recognition Works if constraints on writer, e.g. palm pilot

11. Other document processing Not just for text... Examples: –Engineering document to CAD file –Maps to GIS format –Music score to MIDI representation

12. Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

13. Biometrics Technology for identification –Finger/palm print –Iris –Face

14. Fingerprints Minutae – spits and merges of ridges

15. Face Identification Not quite reliable yet. –Performance degrades rapidly with uncontrolled lighting, facial expression, and size of database Several companies exist: –Visionics (Rockfeller Univesity spin-off) –EyeMatic (USC spin-off) –Miros (MIT spin-off) –Banque-Tec Intl (Australia) –C-VIS Computer Vision (Germany) –LAU Technologies Commercial systems installed in London and Brazil to catch criminals

16. Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

17. Management of images on computers Compression – reducing storage size needed for images Watermarking – Protecting copyright Microsoft, Bell Labs, NEC, etc. Visible watermark

18. Photo-manipulation Adobe Photoshop, Corel PhotoPaint, Pixami, PhotoIQ, etc. –More automatic features

19. Searching Digital Image Collections Large collections of images exist –Corbis 67 million images –Getty 70 million stock photography images –AP collects 1000s of digitized images per day Search methods are inadequate –Rely on captions and colors IBM’s Query by Image Content (QBIC) system

20. Existing and Emerging Technology 1. Document scanning 2. Biometrics 3. Management of images on computers 4. Other: manufacturing, military, games, etc

21. Inspection for Manufacturing Occum – inspection of printed circuit boards ($100M / year) Cognex – Do-it-yourself toolkits for inspection (400 employees)

22. Automatic Target Recognition (ATR) Finding mines, tanks, etc. Billion dollar a year industry –Martin-Lockheed, TSR, Northrup-Grumman, other aerospace contractors. Various types of imagery: –Synthetic Aperture Radar (SAR), Sonar, hyper- spectral imagery (more than 3 colors)

23. Aerial Photo Interpretation / Automated Cartography Classification of land-use: forest, vegetation, water Identification of man-made objects: buildings, roads, etc

24. Better Security Cameras Cameras that are responsive to the environment –Track and zoom on moving objects –Automatic adjustment of contrast

25. Human-Computer Interaction Computer games that involve interaction with user Intelligent teleconferencing

26. Medical imagery Medical image libraries for study and diagnosis Image overlay to guide surgeons

27. History 1980’s ~100 companies – manufacturing applications mostly Early 1990’s less than 10 companies Late 1990’s ~100 companies – face recognition, intelligent teleconferencing, inspection, digital libraries

28. Computer Vision and Image Processing Research

29. Image processing Computer vision chair, face, shape, etc. “Symbolic” descriptionimage

30. 1. Image Processing: Filtering

31. 2. Image Processing: Compression Lossless – No loss in quality, gif, tiff Lossy – Original image cannot be reconstructed, jpeg

32. 3. Image Processing: Watermarking Information hiding –Protecting Copyright

33. 4. Image Processing: Transformation Transforming image can make it easier to analyze Wavelet transform of image

34. Decomposition in Resolution/Frequency fine coarseintermediate

35. Wavelet Decomposition Vertical subbands (LH)

36. Wavelet Decomposition Horizontal subbands (HL)

37. 1. Computer Vision: 3D Shape Reconstruction Use images to build 3D model of object or site 3D site model built from laser range scans collected by CMU autonomous helicopter

38. 2. Computer Vision: To guide Motion Visually guided locomotion –robotic vehicles Visually guided manipulation –Hand-eye coordination CMU NavLab II

39. 3. Computer Vision: Recognition and Classification

40. Challenges in Object Recognition 245 267 234 142 22 28 38 121 156 187 98 73 32 12 123 21 21 38 209 237 121 99 87 59 197 216 244

41. Challenges in Object Detection Intra-class variation

42. Lighting variation

43. Geometric variation

44. Simpler Problem: Classification Fixed size input Fixed object size, orientation, and alignment “Object is present” (at fixed size and alignment) “Object is NOT present” (at fixed size and alignment) Decision

45. 1) Apply Local Operators f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723

46. 2) Look-Up Probabilities f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723 P 1 ( #5710, 0, 0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214, 0, 1 | obj) = 0.57 P 1 ( #3214, 0, 1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = 0.19

47. Probabilities Estimated Off-Line f 1 (0, 0) = #567H 1 (#567, 0, 0) = H 1 (567, 0, 0) + 1 f k (n, m) = #350H k (#350, 0, 0) = H k (#350, 0, 0) + 1 P 1 (#567, 0, 0) =  H 1 (#i, 0, 0) H 1 (#567, 0, 0) P k (#350, 0, 0) =  H k (#i, 0, 0) H k (#350, 0, 0)

48. 3) Make Decision P 1 ( #5710, 0, 0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214, 0, 1 | obj) = 0.57 P 1 ( #3214, 0, 1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = 0.19 0.53 * 0.57 *... * 0.83 0.56 * 0.48 *... * 0.19 >

49. Overall Algorithm f 1 (0, 1) = #3214 f 1 (0, 0) = #5710 f k (n, m) = #723 P 1 ( #5710,0,0 | obj) = 0.53 P 1 ( #5710, 0, 0 | non-obj) = 0.56 P 1 ( #3214,0,1 | obj) = 0.57 P 1 ( #3214,0,1 | non-obj) = 0.48 P k ( #723, n, m | obj) = 0.83 P k ( #723, n, m | non-obj) = 0.19 0.53 * 0.57 *... * 0.83 0.56 * 0.48 * … * 0.19

50. Detection: Apply Classifier Exhaustively Search in position Search in scale

51. View-based Classifiers Face Classifier #1 Face Classifier #2 Face Classifier #3

52. 2 classifiers trained for faces. 8 classifiers trained for cars.

53. Training Classifiers Cars: 300-500 images per viewpoint Faces: 2,000 images per viewpoint ~1,000 synthetic variations of each original image –background scenery, orientation, position, frequency 2000 non-object images –Samples selected by bootstrapping Minimization of classification error on training set –AdaBoost algorithm (Freund & Shapire ‘97, Shapire & Singer ‘99) Iterative method Determines weights for samples

61. Applications of Face Detection Automatic red-eye removal from photographs Automatic color balancing in photo- finishing Intelligent teleconferencing Component in face identification system

62. Difficulty Increases with Complexity of Object 2D vs. 3D Specific objects – e.g. my coffee mug A category of objects – e.g. all coffee mugs Amount of intra-category variation –Rigid or semi-rigid structure, e.g. face –Articulated objects, e.g. human body –Functionally defined objects, e.g. chairs

63. Summary: Image Processing & Computer Vision Not as mature as speech recognition –Technology not as reliable –Fewer companies, fewer products Success on limited problems, e.g., documents More applicable to fault tolerant problems Technology will grow –Emergence of digital camera –Improved methods