1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Using Image Data in Your Research Kenton McHenry, Ph.D. Research Scientist

2. Image and Spatial Data Analysis Group

3. Research & Development Cyberinfrastructure: Software development for the sciences (and industry) Computer Vision: Information from images High Performance Computing: Software that scales with regards to computation and data

4. Image and Spatial Data Analysis Group Content Based Retrieval Search in digitized collections Document segmentation Authorship 3D models Automatic Image Annotation Assign keywords as metadata Tracking 3D Reconstruction Image Stitching

5. Image and Spatial Data Analysis Group Digital Preservation Access to data content independent of format Access to software functionality independent of distribution Information loss evaluation Document similarity Environmental Modeling Workflows Heterogeneous data sources Data Exploration Data mining eScience

6. Goals for Today A high level understanding of what Computer Vision is and how YOU might use it. A sense of what is currently possible A sense of how these things break A sense of what might be possible A sense of what is pure science fiction! The looming opportunity in “Big Data” A little bit of hands on experience

7. Computer Vision Books: D. Forsyth, J. Ponce, “Computer Vision: A Modern Approach”, Pearson, 2011. R. Szeliski, “Computer Vision: Algorithms and Applications”, http://szeliski.org/Book, 2010. http://szeliski.org/Book CS 543: Computer Vision (UIUC) Derek Hoiem, Ph.D. http://www.cs.illinois.edu/class/sp12/cs543

8. Computer Vision [Hoiem, 2012]

9. Computer Vision Make a computer understand images and video What kind of scene? Are there cars? Where are the cars? Is it day or night? What is the ground made of? How far is the building? [Hoiem, 2012]

10. Raster Images 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 [Hoiem, 2012]

11. Image Creation Light emitted Sensor Lens Fraction of light reflects into camera [Hoiem, 2012]

12. Image Creation Light(s) Position Strength Geometry Color Surface(s) Orientation Color Material Nearby surfaces Sensor Lens Aperture Exposure Resolution Light emitted Sensor Light reflected to camera [Hoiem, 2012]

13. Surfaces: Reflected Light incoming light specular reflection incoming light diffuse reflection absorption incoming light [Hoiem, 2012]

14. Surface: Reflected Light

15. Surfaces: Orientation 1 2 I x =  x LN x [Hoiem, 2012]

16. Surfaces light sourcetransparency light source refraction [Hoiem, 2012]

17. Surfaces λ1λ1 light source λ2λ2 fluorescence

18. Surfaces t=1 light source t>1 phosphorescence [Hoiem, 2012]

19. Surfaces λ light source subsurface scattering [Hoiem, 2012]

20. Light Human Luminance Sensitivity Function [Hoiem, 2012]

21. Light [Hoiem, 2012]

22. Light

23. [GIMP Demo]

24. Sensors Long (red), Medium (green), and Short (blue) cones, plus intensity rods [Hoiem, 2012]

25. Sensors [Hoiem, 2012]

26. Sensors R G B [Hoiem, 2012]

27. Sensors: Perspective Projecting a 3D world onto a 2D plane Parallel lines disappear at vanishing points Sizes appear smaller further away

28. Surface Interactions! [Hoiem, 2012]

29. Surface Interactions [Hoiem, 2012]

30. Surface Interactions [Hoiem, 2012]

31. Surfaces: Interactions

32. Surface Interactions [Hoiem, 2012]

33. Raster Images 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 [Hoiem, 2012] image(234, 452) = 0.58

34. Individual Pixels [Hoiem, 2012]

35. Neighborhoods of Pixels For nearby surface points most factors do not change much Local differences in brightness [Hoiem, 2012]

36. Neighborhoods of Pixels [Hoiem, 2012]

37. Neighborhoods of Pixels [Hoiem, 2012]

38. Neighborhoods of Pixels [Hoiem, 2012]

39. Changes in Intensity Changes in albedo Changes in surface normal Changes in distance [Hoiem, 2012]

40. Computer Vision Make a computer understand images and video Lots of variables are involved in the creation of an image/frame Variables are not independent and interact The problem is underconstraned i.e. multiple scenes can result in the same image

41. Optical Illusions

44. Vision is Really Hard! Vision is an amazing feat of natural intelligence More human brain devoted to vision than anything else [Hoiem, 2012]

45. State of the Art From 1960’s to present…

46. Barcodes Optical machine readable representation of data 1950’s http://en.wikipedia.org/wiki/Barcode

47. Optical Character Recognition (OCR) Digit recognition, AT&T labs http://www.research.att.com/~yannhttp://www.research.att.com/~yann/ Technology to convert scanned documents to ASCII text If you have a scanner, it probably came with OCR software License plate readers http://en.wikipedia.org/wiki/Automatic_number_plate_recognition [Hoiem, 2012]

48. Biometrics Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely http://www.sensiblevision.com/ http://www.sensiblevision.com/ [Hoiem, 2012]

49. Face detection Many new digital cameras now detect faces Canon, Sony, Fuji, … [Hoiem, 2012]

50. Medical imaging 3D imaging, MRI, CT [Hoiem, 2012], http://en.wikipedia.org/wiki/3D_ultrasound

51. The Matrix movies, ESC Entertainment, XYZRGB, NRC 3D Reconstruction

52. Pirates of the Carribean, Industrial Light and Magic Motion capture [Hoiem, 2012]

53. Image Stitching NASA'S Mars Exploration Rover Spirit NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007. [Hoiem, 2012]

54. Industry Vision-guided robots position nut runners on wheels [Hoiem, 2012]

55. Sports http://www.sportvision.com/video.html [Hoiem, 2012]

56. Object Recognition Point & FindPoint & Find, Nokia, Google GogglesNokiaGoogle Goggles [Hoiem, 2012] LaneHawk by EvolutionRobotics

57. Human Computer Interaction Object Recognition: http://www.youtube.com/watch?feature=iv&v=fQ59dXOo63ohttp://www.youtube.com/watch?feature=iv&v=fQ59dXOo63o 3D Reconstruction: http://www.youtube.com/watch?v=7QrnwoO1-8Ahttp://www.youtube.com/watch?v=7QrnwoO1-8A Robot: http://www.youtube.com/watch?v=w8BmgtMKFbYhttp://www.youtube.com/watch?v=w8BmgtMKFbY [Hoiem, 2012]

58. Driving Oct 9, 2010. "Google Cars Drive Themselves, in Traffic". The New York Times. John Markoff"Google Cars Drive Themselves, in Traffic"The New York Times June 24, 2011. "Nevada state law paves the way for driverless cars". Financial Post. Christine Dobby"Nevada state law paves the way for driverless cars"Financial Post Aug 9, 2011, "Human error blamed after Google's driverless car sparks five-vehicle crash". The Star (Toronto)"Human error blamed after Google's driverless car sparks five-vehicle crash" [Hoiem, 2012]

59. State of the Art Remember vision is hard! Most vision applications are “quirky”.

60. Image and Spatial Data Analysis Group http://isda.ncsa.illinois.edu Questions?