1. Recognizing and Tracking Human Action Josephine Sullivan and Stefan Carlsson

2. Define Tracking

3. Traditional tracking Kalman Filters Condensation HMM Matching articulated 3d models Similarities? Problems?

4. New approach What is the difference between tracking and recognition? Assume Pose recognition and activity recognition are equivalent. Now track activity by repeating recognition of key frames

5. Discussion: reasons for previous approach Why the distinction between tracking and recognition? Applications? –Projectile tracking –Motion capture

6. Object descriptors Embedding global data in local descriptors Order Structure Shape context

7. Order Structure Problem: find correspondence between deformed shapes Solution –Sample points on contour –Describe shape using order structure Order of points and intersections of tangent lines

8. Order Structure Many transformations preserve order structure –Superset of Affine and Projective transformations –Encodes perceptual similarity Encodes properties of point sets, lines, and combinations of points and lines. Descriptor for Point sets - orientation Set {a,b,c} has + orientation if traversing them in order means anti-clockwise rotation

9. Order Structure Descriptor for Sets of lines –Uses: points and lines are projectively dual –p - homogeneous coord’s for a point –q - oriented homogeneous line coord’s for line thru p, then: q T p = 0 –q = (a,1,b) where ax+y+b = 0. –Order type for a set of 3 lines is then

10. Order Structure Descriptor for combinations of points and lines –Oriented coordinates => every line has a direction Assign a left-right position for every point w.r.t every line Unique order structure for arbitrary set of points Order structure for a set characterized by an index q i = line p j = point

11. Order Structure Algorithm Voting matrix

12. Order Structure Perceptual similarity example: human pose

13. Shape Context descriptor Sample points from edges in image Each point’s descriptor is a histogram of the relative coordinates of all other points.

14. Action Recognition using Key Frames Deciding images are related –p a i and p b i are coordinates of corresponding points in images A and B. –T is class of transformations that define relation between A and B. (known a priori) –Matching Distance General case Using pure translation

15. Action recognition using Key Frames 30 second tennis sequence “Coarse” automatic tracking Edge detection done on upper half of player –No deletion of background edges Selected a key frame and computed matching score wrt. each other frame. 9 local minima shown, each the start of a forehand stroke.

16. Action recognition using Key Frames

17. Tracking Point transferral –Each key frame is marked manually –For each point in key frame, a subset of points in the image are chosen, and a translation is estimated. Point in keyframe R Simple local translation Point corresponding to P k R in image I t

18. Updating the Voting Matrix Extra information to improve accuracy Use “standard tracker” for head and body localization. (Brand, “Shadow Puppetry”) Set V(p i R, p j t ) = 0 if the points aren’t close to the corresponding lines in corresponding matched head/body quadrangles.

19. Further constraints Want to enforce similar arrangement of interior points in images that are matched to key frames Also incorporate intensity around points Monte-Carlo smoothing is used to correct outlying points

20. Tracking using Shape Context Mori & Malik Very similar technique, using shape context descriptor Very clear that frames are processed independently Tested on standard data

21. Tracking w/Shape Context Movie

22. Discussion & Questions Results - how effective? Effect of rate of motion? Efficiency of “closed loop system”? No need for background subtraction? Flexibility to multiple actions? Do they give a specific order to key frames? Is the coarse tracking too simple? What about poses facing away from camera?