1. Depth and Motion Discontinuities Stan Birchfield Ph.D. oral defense Stanford University January 1999

2. PHOTOMETRIC GEOMETRIC Discontinuities intensity, color, texture,... DepthMotion camera surface camera surface

3. Fundamental to image understanding Object boundaries (well-defined) Assumption of continuity Occlusion Importance of Discontinuities camera 1camera 2 object

4. Motivation segmentation classification tracking camera control video retrieval detection IMAGESLOW-LEVEL FILTERSHIGH-LEVEL TASKS

5. Evidence for Discontinuities LOCAL –T-junctions [Parida et al. 1997, Ruzon & Tomasi 1999] –Occlusion [Toh & Forrest 1990, Wixson 1993] –Multi-modal depth or motion [Spoerri & Ullman 1987, Little & Gillett 1990] –Changes in intensity, color, texture, … [Canny 1986, Wang & Binford 1994] GLOBAL –“Continuity of discontinuities” [Marr & Poggio 1979] –Changes in depth or motion field [ Thompson et al. 1982, Birchfield & Tomasi 1998]

6. Complications from Sampling Amount of discontinuity Curvature ?

7. Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

8. Depth Discontinuities from a Stereo Pair Use dynamic programming to minimize: intensity L R disparity lamp wall pixel dissimilarity discontinuity penalty epipolar constraint

9. Discontinuity Penalty u(l ) i penalty occlusion length u(3) = 3u(1) u(3) < 3u(1) BAD GOOD l i L R L R BAD disparity pixel intensity

10. Stereo Algorithm Matches pixels directly with convex penalty function (no preprocessing or windows) Handles large untextured regions Solves sampling problem Handles slanted surfaces Is fast

11. Handling Untextured Regions Our constraint: Depth discontinuities lie to a particular side of intensity edge Discontinuities in left (right) scanline lie to the left (right) of intensity edge Assumption: Depth discontinuities are accompanied by intensity edge Naïve constraint: Depth discontinuities lie near intensity edge Intensity edges (x-derivative with low threshold) With naïve constraint With our constraint

12. Discontinuity Lies to the Left of an Edge near object far object left camera right camera Obvious matches Physical setup Matches violate assumption Matches are consistent with assumption edge depth discontinuity

13. The Problem of Image Sampling Absolute difference Our measure Left scanlineRight scanline

14. A Dissimilarity Measure That is Insensitive to Image Sampling d(x L,x R ) xLxL xRxR d(x L,x R ) = min{d(x L,x R ),d(x R,x L )}Our dissimilarity measure:

15. Analysis of Dissimilarity Measure Concave/convex regions: measure is guaranteed to work Inflection points: measure works in practice when lens is defocused to remove aliasing (inflection points  linear {convex, concave}) intensity function

16. Analysis of Dissimilarity Measure our measure absolute difference maximum average T c = 1 fcfc cutoff frequency As a function of lens defocus At a hypothesized disparity of 10 pixels aliasing { defocus

17. Speeding Computation D isparity map occlusion depth discontinuity Pruning bad nodes Standard Ours maximum disparity time Fast postprocessing TWO STEPS: 1. Reliable overruns unreliable 2. Background overruns foreground RIGHT LEFT

18. Handling Slanted Surfaces Independent scanlines: no coherence Postprocessing: forbid propagation if disparity changes by just one level disparity column

19. Results

20. More Results [Images from JISCT data set]

21. Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

22. Motion Discontinuities from a Sequence Similar to stereo (matching pixels to fit piecewise-smooth function) But different (2D search, non-rigid transformation, many images) How far can we get using sparse features?

23. Motion Video

24. Difficulties of Image Sequences Instantaneous velocity from sampled positions Accumulation of evidence Frame of reference, motion model time position ? feature position time feature background threshold position time feature 2 feature 1 background threshold

25. Image Strain

26. Motion Algorithm Select and track features Group Trace boundaries between groups

27. Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

28. Why Maximum Flow? Our stereo algorithm (Dynamic programming) –Good results on difficult images –Fast –But, suboptimal (processes rows, then columns independently) Newer algorithms (Maximum flow) –Set up graph, find maximum flow --- minimum cut yields disparity or motion map –Able to look at whole image

29. Minimizing a 2D Cost Function Minimize: disparity 2D: GLOBAL disparity pixel ? 1D: Global u(l ) p,q Discontinuity penalty: l p,q minimum cut = disparity surface solves LOCAL Local (GOOD) (BAD) 

30. Maximum Flow for Stereo and Motion Global algorithm [Roy & Cox 1998, Ishikawa & Geiger 1998] Local algorithm [Boykov et al. 1998] penalty BAD discontinuity amount GOOD

31. Challenges for Maximum Flow … and slant of table With our intensity variation constraint and large penalty With small penaltyWith large penalty … and slant of surfaces

32. Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

33. Problem TILT PAN ZOOM CHALLENGES: * rotation * multiple people * zoom APPLICATIONS: * video conferencing * distance learning

34. Previous Methods 3D RotationMoving people in background Skin color: N? Background subtraction: YN Template: NY Contour: YN

35. Head Tracking Algorithm MODEL (x,y)  Ellipse: vertical aspect ratio = 1.2  state s = (x,y,  ) ellipse in frame t OUTPUT color module Model Current Intersection SkinHair LOCAL SEARCH gradient module gradient ellipse normal ellipse in frame t INPUT frame t-1

36. Evaluation of Head Tracker 1. Tracks head in real time on standard hardware 2. Insensitive to - full 360-degree out-of-plane rotation - arbitrary camera movement (including zoom) - multiple moving people - severe but brief occlusion - hair/skin color, hair length, facial hair, glasses

37. Headtracker Video

38. Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

39. Contributions Precise, unified definitions and relationship Pixel-to-pixel stereo algorithm [ICCV 1998] –good results on difficult images intensity variation constraint dissimilarity measure [PAMI 1998] preserving slant –fast (pruning search nodes, postprocessor) Motion discontinuities (sparse features, image strain, grouping algorithm) Maximum flow techniques (edge weights for global algorithm, smoothness, left border ; applied to motion) Elliptical head tracker [Asilomar 1997, CVPR 1998] –advanced state of the art by tracking people undergoing 3D rotation in front of a dynamic background

41. Appendix: Extra slides

42. Comparison of Our Algorithm with Maxflow Our algorithm Local maxflow algorithm

43. Depth Discontinuity Proposition 1: A depth discontinuity is a point in the image plane whose projection ray grazes a surface in the world. object image plane focal point Depth discontinuities

44. Definition of Jump The jump of a discontinuity is the smallest  >0 such that there is a  >0 for which, if ||x-x 0 ||  x f(x) x0x0   jump

45. Comparison with Intensity Edges Original imageIntensity edgesDepth discontinuities More closely tied to actual object boundaries Not distracted by change in albedo (reflectance) change in lighting (e.g., shadows)

46. Comparison with Layers Original imageMotion discontinuities

47. Comparison with Segmentation A plausible segmentation Depth discontinuities Original image Well-defined (not task-dependent) Curves are not necessarily closed

48. Epipolar Constraint Left cameraRight camera world point center of projection epipolar plane epipolar line

49. Comparison of Dissimilarity Measure with Other Methods absolute difference our measure spurious matches subpixel resolution Computing time

50. Propagating Information Between Scanlines Calculate reliabilities & threshold Disparities: [ 4 4 4 4 4 7 7 7 2 ] T Reliabilities: [ 5 5 5 5 5 3 3 3 1 ] T unreliable < reliable < very reliable Propagate very reliable when –neighbor unreliable, OR –disparity(neighbor) > disparity(pixel) + 2 Repeat in x-direction Fast (30% increase instead of 800%)

51. Features on Freethrow

52. Maximum Flow Finds greatest shipping rate from source to sink w/o violating capacity constraints Flow network G=(V,E) is directed graph with edge capacities c(u,v) > 0, source s, and sink t Flow f:E R Value of flow |f| =  f(s,v) Find f* = arg max |f| Cut (S,T) is partition of V s.t. s in S and t in T capacity of cut c(S,T) Max-flow min-cut theorem: There exists a cut (S,T) s.t. |f*| = c(S,T)

53. Blowup of Meter Global algorithm Local algorithm

54. Maximum Flow and Occlusion [from University of Tsukuba, Japan]

55. Maximum Flow and Left Image Border