1. Algorithms and Applications in Computer Vision
Lihi Zelnik-Manor

2. Today Administration “What is vision?” Schedule Introductions
Image formation

3. Prerequisites What I expect you to already know:
A good working knowledge of MATLAB programming (or willingness and time to pick it up quickly!)
Linear algebra
Vector calculus

4. Books
Rick Szeliski’s book: Computer Vision: Algorithms and applications
A secondary book: Forsyth and Ponce, Computer Vision: A Modern Approach.

5. Primary Book

6. Secondary Book

7. Matlab
Problem sets and projects will involve Matlab programming (you are free to use alternative packages).

8. Grading There will be three equal components to the course grade
Three problem sets
Final project
Class participation

9. Problem sets Pset1 – Image formation
Pset2 – Feature detection and matching
Pset3 – Image alignment and flow
Pset4 – To be determined
Can discuss, but must submit individual work

10. Class Participation Class participation includes showing up
Ask wise questions
Ask trivial questions
Correct me when I’m wrong or inaccurate

11. Course goals….
You’ll know something about computer vision

12. What is computer vision?
Done?

13. What is computer vision?
Automatic understanding of images and video
Computing properties of the 3D world from visual data (measurement)
Algorithms and representations to allow a machine to recognize objects, people, scenes, and activities. (perception and interpretation)

14. Vision for measurement
Multi-view stereo for community photo collections
Real-time stereo
Structure from motion
NASA Mars Rover
Pollefeys et al.
Goesele et al.
Slide credit: L. Lazebnik

15. Vision for perception, interpretation
Objects
Activities
Scenes
Locations
Text / writing
Faces
Gestures
Motions
Emotions…
amusement park
sky
The Wicked Twister
Cedar Point
Ferris wheel
ride
ride
12 E
Lake Erie
water
ride
tree
tree
people waiting in line
people sitting on ride
umbrellas
tree
maxair
carousel
deck
bench
tree
pedestrians 15

16. Artificial intelligence
Related disciplines
Artificial intelligence
Graphics
Machine learning
Computer vision
Cognitive science
Image processing
Algorithms

17. Vision and graphics Inverse problems: analysis and synthesis. Images
Model
Graphics
Inverse problems: analysis and synthesis.

18. Why vision? As image sources multiply, so do applications
Relieve humans of boring, easy tasks
Enhance human abilities: human-computer interaction, visualization
Perception for robotics / autonomous agents
Organize and give access to visual content

19. Why vision? Images and video are everywhere! Personal photo albums
Movies, news, sports
Surveillance and security
Medical and scientific images
Slide credit; L. Lazebnik

20. Again, what is computer vision?
Mathematics of geometry of image formation?
Statistics of the natural world?
Models for neuroscience?
Engineering methods for matching images?
Science Fiction?

21. Vision Demo? we’re not quite there yet…. Terminator 2
Clips: terminator 2, enemy of the state (from UCSD “Fact or Fiction” DVD)
we’re not quite there yet….
Terminator 2

22. Every picture tells a story
Goal of computer vision is to write computer programs that can interpret images

23. Can computers match (or beat) human vision?
If you can write a formula for it, computers can excel
Computer vision can’t solve the whole problem (yet), so breaks it down into pieces.
Many of the pieces have important applications.
Yes and no (but mostly no!)
humans are much better at “hard” things
computers can be better at “easy” things

24. Human perception has its shortcomings…
Example where humans make mistakes that computers can avoid
Sinha and Poggio, Nature, 1996

25. Copyright A.Kitaoka 2003

26. Current state of the art
The next slides show some examples of what current vision systems can do

27. Earth viewers (3D modeling)
Image from Microsoft’s Virtual Earth
(see also: Google Earth)

28. Photosynth http://labs.live.com/photosynth/
Based on Photo Tourism technology developed by Noah Snavely, Steve Seitz, and Rick Szeliski

29. Photo Tourism overview
Photo Explorer
Scene reconstruction
Input photographs
Relative camera positions and orientations
Point cloud
Sparse correspondence
System for interactive browsing and exploring large collections of photos of a scene.
Computes viewpoint of each photo as well as a sparse 3d model of the scene.

30. Photo Tourism overview

31. Optical character recognition (OCR)
Technology to convert scanned docs to text
If you have a scanner, it probably came with OCR software
Digit recognition, AT&T labs
License plate readers

32. Face detection Many new digital cameras now detect faces
Why would this be useful? Main reason is focus. Also enables “smart” cropping.
Many new digital cameras now detect faces
Canon, Sony, Fuji, …

33. Smile detection?
Sony Cyber-shot® T70 Digital Still Camera

34. Face alignment
Ira Kemelmacher-Shlizerman, Rahul Garg, Steve Seitz,

35. Object recognition (in supermarkets)
LaneHawk by EvolutionRobotics
“A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk,you are assured to get paid for it… “

36. Face recognition
Who is she?

37. Vision-based biometrics
“How the Afghan Girl was Identified by Her Iris Patterns” Read the story

38. Login without a password…
Face recognition systems now beginning to appear more widely
Fingerprint scanners on many new laptops, other devices

39. Object recognition (in mobile phones)
This is becoming real:
Microsoft Research
Point & Find, Nokia
SnapTell.com (now amazon)

40. Snaptell

41. Nokia Point and Tell…

42. Special effects: shape capture
The Matrix movies, ESC Entertainment, XYZRGB, NRC

43. Special effects: motion capture
Pirates of the Carribean, Industrial Light and Magic

44. Sports Sportvision first down line
Nice explanation on

45. Slide content courtesy of Amnon Shashua
Smart cars
Mobileye
Vision systems currently in high-end BMW, GM, Volvo models
By 2010: 70% of car manufacturers.
Video demo
Slide content courtesy of Amnon Shashua

46. Slide content courtesy of Amnon Shashua
Smart cars
Mobileye
Vision systems currently in high-end BMW, GM, Volvo models
By 2010: 70% of car manufacturers.
Video demo
Slide content courtesy of Amnon Shashua

47. Vision-based interaction (and games)
Digimask: put your face on a 3D avatar.
Nintendo Wii has camera-based IR tracking built in. See Lee’s work at CMU on clever tricks on using it to create a multi-touch display!
“Game turns moviegoers into Human Joysticks”, CNET
Camera tracking a crowd, based on this work.

48. Vision in space Vision systems (JPL) used for several tasks
NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.
Vision systems (JPL) used for several tasks
Panorama stitching
3D terrain modeling
Obstacle detection, position tracking
For more, read “Computer Vision on Mars” by Matthies et al.

49. NASA’s Mars Spirit Rover
Robotics
NASA’s Mars Spirit Rover

50. Medical imaging Image guided surgery 3D imaging Grimson et al., MIT
MRI, CT

51. Current state of the art
You just saw examples of current systems.
Many of these are less than 5 years old
This is a very active research area, and rapidly changing
Many new apps in the next 5 years
To learn more about vision applications and companies
David Lowe maintains an excellent overview of vision companies

52. Syllabus / Schedule Image Formation Color Image Filtering
Image Formation
Color
Image Filtering
Pyramids & Regularization
Feature Detection and Matching
Geometric Alignment
Geometric Image Stitching
Photometric Image Stitching
Stereo
Optic Flow
Dense Motion Models
Shape from motion
Segmentation
Texture

53. And now, who are you? What do you expect to get out of this class?
Previous experience in vision, learning, graphics?
Research agenda?
(Project topics?)

54. Slide Credits Slides 14-21, 55-66: Kristen Grauman
Slides 23-40,43-52: Steve Seitz
and others, as marked…