1. Stockman MSU Fall 20091 Computing Motion from Images Chapter 9 of S&S plus otherwork.

2. Stockman MSU Fall 20092 General topics Low level change detection Region tracking or matching over time Interpretation of motion MPEG compression Interpretation of scene changes in video Understanding human activites

3. Stockman MSU Fall 20093 Motion important to human vision

4. Stockman MSU Fall 20094 What’s moving: different cases

5. Stockman MSU Fall 20095 Image subtraction Simple method to remove unchanging background from moving regions.

6. Stockman MSU Fall 20096 Change detection for surveillance

7. Stockman MSU Fall 20097 Change detection by image subtraction

8. Stockman MSU Fall 20098 What to do with regions of change? Discard small regions Discard regions of non interesting features Keep track of regions with interesting features Track in future frames from motion plus component features

9. Stockman MSU Fall 20099 Some effects of camera motion that can cause problems

10. Stockman MSU Fall 200910 Motion field

11. Stockman MSU Fall 200911 FOE and FOC Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.

12. Stockman MSU Fall 200912 Gaming using a camera to recognize the player’s motion Decathlete game

13. Stockman MSU Fall 200913 Decathlete game Cheap camera replaces usual mouse for input Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.

14. Stockman MSU Fall 200914 Motion detection input device Running (hands) Jumping (hands)

15. Stockman MSU Fall 200915 Motion analysis controls hurdling event (console) Top left shows video frame of player Middle left shows motion vectors from multiple frames Center shows jumping patterns

16. Stockman MSU Fall 200916 Related work Motion sensed by crude cameras Person dances/gestures in space System maps movement into music Creative environment? Good exercise room?

17. Stockman MSU Fall 200917 Computing motion vectors from corresponding “points” High energy neighborhoods are used to define points for matching

18. Stockman MSU Fall 200918 Match points between frames Such large motions are unusual. Most systems track small motions.

19. Stockman MSU Fall 200919 Requirements for interest points Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.

20. Stockman MSU Fall 200920 Interest = minimum directional variance Used by Hans Moravec in his robot stereo vision system. Interest points were used for stereo matching.

21. Stockman MSU Fall 200921 Detecting interest points in I1

22. Stockman MSU Fall 200922 Match points from I1 in I2

23. Stockman MSU Fall 200923 Search for best match of point P1 in nearby window of I2 For both motion and stereo, we have some constraints on where to search for a matching interest point.

24. Stockman MSU Fall 200924 Motion vectors clustered to show 3 coherent regions All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin) Motion coherence: points of same object tend to move in the same way

25. Stockman MSU Fall 200925 Two frames of aerial imagery Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

26. Stockman MSU Fall 200926 Can code frame N+d with displacments relative to frame N for each 16 x 16 block in the 2 nd image find a closely matching block in the 1 st image replace the 16x16 intensities by the location in the 1 st image (dX, dY) 256 bytes replaced by 2 bytes! (If blocks differ too much, encode the differences to be added.)

27. Stockman MSU Fall 200927 Frame approximation Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)

28. Stockman MSU Fall 200928 Best matching blocks between video frames N+1 to N (motion vectors) The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2! Best matches from 2 nd to first image shown as vectors overlaid on the 2 nd image. (Work by Dina Eldin.)

29. Stockman MSU Fall 200929 Motion coherence provides redundancy for compression MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects

30. Stockman MSU Fall 200930 MPEG represents blocks that move by the motion vector

31. Stockman MSU Fall 200931 MPEG has ‘I’, ‘P’, and ‘B’ frames

32. Stockman MSU Fall 200932 Computing Image Flow

33. Stockman MSU Fall 200933

34. Stockman MSU Fall 200934 Assumptions

35. Stockman MSU Fall 200935 IMAGE FLOW EQUATION 1 of 2

36. Stockman MSU Fall 200936 Image flow equation 2 of 2

37. Stockman MSU Fall 200937 Aperture problem

38. Stockman MSU Fall 200938 Solving flow by propagation of constraints

39. Stockman MSU Fall 200939 Info at corner constrains the flow along both edges Solve constraints using contraint propagation or differential equation with boundary conditions.

40. Stockman MSU Fall 200940 Tracking several objects Use assumptions of physics to compute multiple smooth paths. (work of Sethi and R. Jain)

41. Stockman MSU Fall 200941

42. Stockman MSU Fall 200942 Tracking in images over time

43. Stockman MSU Fall 200943 General constraints from physics

44. Stockman MSU Fall 200944 Other possible constraints Background statistics stable Object color/texture/shape might change slowly over frames Might have knowledge of objects under survielance Objects appear/disappear at boundary of the frame

45. Stockman MSU Fall 200945

46. Stockman MSU Fall 200946 Sethi-Jain algorithm

47. Stockman MSU Fall 200947

48. Stockman MSU Fall 200948 Total smoothness of m paths

49. Stockman MSU Fall 200949 Greedy exchange algorithm

50. Stockman MSU Fall 200950 Example data structure Total smoothness for trajectories of Figure 9.14

51. Stockman MSU Fall 200951 Example of domain specific tracking (Vera Bakic) Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback- control loop.

52. Stockman MSU Fall 200952 Segmentation of videos/movies Segment into scenes, shots, specific actions, etc.

53. Stockman MSU Fall 200953 Types of changes in videos

54. Stockman MSU Fall 200954 Anchor person scene at left Street scene for news story Scene break From Zhang et al 1993 How do we compute the scene change?

55. Stockman MSU Fall 200955 Histograms of frames across the scene change Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.

56. Stockman MSU Fall 200956 Heuristics for ignoring zooms

57. Stockman MSU Fall 200957 American sign language example

58. Stockman MSU Fall 200958 Example from Yang and Ahuja

59. Stockman MSU Fall 200959

60. Stockman MSU Fall 200960