1. COS 429: Computer Vision Thanks to Chris Bregler

2. COS 429: Computer Vision Instructor: Szymon Rusinkiewicz smr@cs.princeton.eduInstructor: Szymon Rusinkiewicz smr@cs.princeton.edu TA: Nathaniel Dirksen ndirksen@cs.princeton.eduTA: Nathaniel Dirksen ndirksen@cs.princeton.edu Course web page http://www.cs.princeton.edu/courses/cs496/Course web page http://www.cs.princeton.edu/courses/cs496/

3. What is Computer Vision? Input: images or videoInput: images or video Output: description of the worldOutput: description of the world But also: measuring, classifying, interpreting visual informationBut also: measuring, classifying, interpreting visual information

4. Low-Level or “Early” Vision Considers local properties of an image “There’s an edge!”

5. Mid-Level Vision Grouping and segmentation “There’s an object and a background!”

6. High-Level Vision Recognition “It’s a chair!”

7. Big Question #1: Who Cares? Applications of computer visionApplications of computer vision – In AI: vision serves as the “input stage” – In medicine: understanding human vision – In engineering: model extraction

8. Vision and Other Fields Computer Vision Artificial Intelligence Cognitive Psychology Signal Processing Computer Graphics Pattern Analysis Metrology

9. Big Question #2: Does It Work? Situation much the same as AI:Situation much the same as AI: – Some fundamental algorithms – Large collection of hacks / heuristics Vision is hard!Vision is hard! – Especially at high level, physiology unknown – Requires integrating many different methods – Requires reasoning and understanding: “AI completeness”

10. Computer and Human Vision Emulating effects of human visionEmulating effects of human vision Understanding physiology of human visionUnderstanding physiology of human vision Analogues of human vision at low, mid, and high levelsAnalogues of human vision at low, mid, and high levels

11. Image Formation Human: lens forms image on retina, sensors (rods and cones) respond to lightHuman: lens forms image on retina, sensors (rods and cones) respond to light Computer: lens system forms image, sensors (CCD, CMOS) respond to lightComputer: lens system forms image, sensors (CCD, CMOS) respond to light

12. Low-Level Vision Hubel

13. Retinal ganglion cellsRetinal ganglion cells Lateral Geniculate Nucleus – function unknown (visual adaptation?)Lateral Geniculate Nucleus – function unknown (visual adaptation?) Primary Visual CortexPrimary Visual Cortex – Simple cells: orientational sensitivity – Complex cells: directional sensitivity Further processingFurther processing – Temporal cortex: what is the object? – Parietal cortex: where is the object? How do I get it?

14. Low-Level Vision Net effect: low-level human vision can be (partially) modeled as a set of multiresolution, oriented filtersNet effect: low-level human vision can be (partially) modeled as a set of multiresolution, oriented filters

15. Low-Level Depth Cues FocusFocus VergenceVergence StereoStereo Not as important as popularly believedNot as important as popularly believed

16. Low-Level Computer Vision Filters and filter banksFilters and filter banks – Implemented via convolution – Detection of edges, corners, and other local features – Can include multiple orientations – Can include multiple scales: “filter pyramids” ApplicationsApplications – First stage of segmentation – Texture recognition / classification – Texture synthesis

17. Texture Analysis / Synthesis MultiresolutionOriented Filter Bank Original Image ImagePyramid

18. Texture Analysis / Synthesis OriginalTexture SynthesizedTexture Heeger and Bergen

19. Low-Level Computer Vision Optical flowOptical flow – Detecting frame-to-frame motion – Local operator: looking for gradients ApplicationsApplications – First stage of tracking

20. Optical Flow Image #1 Optical Flow Field Image #2

21. Low-Level Computer Vision Shape from XShape from X – Stereo – Motion – Shading – Texture foreshortening

22. 3D Reconstruction Tomasi+Kanade Debevec,Taylor,Malik Phigin et al. Forsyth et al.

23. Mid-Level Vision Physiology unclearPhysiology unclear Observations by Gestalt psychologistsObservations by Gestalt psychologists – Proximity – Similarity – Common fate – Common region – Parallelism – Closure – Symmetry – Continuity – Familiar configuration Wertheimer

24. Grouping Cues

28. Mid-Level Computer Vision TechniquesTechniques – Clustering based on similarity – Limited work on other principles ApplicationsApplications – Segmentation / grouping – Tracking

29. Snakes: Active Contours Contour Evolution for Segmenting an Artery

30. Birchfeld Histograms

31. Expectation Maximization (EM) Color Segmentation

32. Bayesian Methods Prior probabilityPrior probability – Expected distribution of models Conditional probability P(A|B)Conditional probability P(A|B) – Probability of observation A given model B

33. Bayesian Methods Prior probabilityPrior probability – Expected distribution of models Conditional probability P(A|B)Conditional probability P(A|B) – Probability of observation A given model B Bayes’s Rule P(B|A) = P(A|B)  P(B) / P(A)Bayes’s Rule P(B|A) = P(A|B)  P(B) / P(A) – Probability of model B given observation A Thomas Bayes (c. 1702-1761)

34. Bayesian Methods # black pixels

35. High-Level Vision Human mechanisms: ???Human mechanisms: ???

36. High-Level Vision Computational mechanismsComputational mechanisms – Bayesian networks – Templates – Linear subspace methods – Kinematic models

37. Cootes et al. Template-Based Methods

38. Linear Subspaces

39. Data PCA New Basis Vectors Kirby et al. Principal Components Analysis (PCA)

40. Kinematic Models Optical Flow/Feature tracking: no constraints Layered Motion: rigid constraints Articulated: kinematic chain constraints Nonrigid: implicit / learned constraints

41. Real-world Applications Osuna et al:

42. Real-world Applications Osuna et al:

43. Course Outline Image formation and captureImage formation and capture Filtering and feature detectionFiltering and feature detection Motion estimationMotion estimation Segmentation and clusteringSegmentation and clustering PCAPCA 3D projective geometry3D projective geometry Shape acquisitionShape acquisition Shape analysisShape analysis

44. 3D Scanning

45. Image-Based Modeling and Rendering Debevec et al. Manex

46. Course Mechanics 65%: 4 written / programming assignments65%: 4 written / programming assignments 35%: Final group project35%: Final group project

47. Course Mechanics Book: Introductory Techniques for 3-D Computer Vision Emanuele Trucco and Alessandro VerriBook: Introductory Techniques for 3-D Computer Vision Emanuele Trucco and Alessandro Verri PapersPapers

48. COS 429: Computer Vision Instructor: Szymon Rusinkiewicz smr@cs.princeton.eduInstructor: Szymon Rusinkiewicz smr@cs.princeton.edu TA: Nathaniel Dirksen ndirksen@cs.princeton.eduTA: Nathaniel Dirksen ndirksen@cs.princeton.edu Course web page http://www.cs.princeton.edu/courses/cs496/Course web page http://www.cs.princeton.edu/courses/cs496/