1. Human-Computer Interaction Human-Computer Interaction Tracking Hanyang University Jong-Il Park

2. Tracking in computer vision Definition Problem of generating an inference about the motion of an object given a sequence of images Application  Motion capture  Recognition from motion  Surveillance  Who is doing what?  Eg. Security, HCI(Kinect…)

3. Tracking Establish state of object using time sequence  state could be:  position; position+velocity; position+velocity+acceleration  or more complex, Eg. all joint angles for a person  Biggest problem -- Data Association  which image pixels are informative, which are not? Key ideas  Tracking by detection  if we know what an object looks like, that selects the pixels to use  Tracking through flow  if we know how an object moves, that selects the pixels to use

4. Appearance vs. Flow

5. Tracking by Detection Assume  a very reliable detector (e.g. faces; back of heads)  detections that are well spaced in images (or have distinctive properties)  e.g. news anchors; heads in public Link detects across time  only one - easy  multiple - weighted bipartite matching  but what if one is missing? Better: create abstract tracks  link detects to track  create tracks, reap tracks as required  clean up spacetime paths

7. Tracking by Known Appearance Even if we don’t have a detector Know rectangle in image (n) Want to find corresponding rectangle in (n+1) Search over nearby rectangles  to find one that minimizes SSD error where sum is over pixels in rectangle Application  stabilize players in TV sport

10. Buidling Tracks Start at scattered points in image 1  perhaps corner detector responses For each in image (n), compute position in (n+1)  as in previous slide Now check tracks  patch in (n+1) should look like an affine transform of patch in 1  Prune bad tracks

13. What if the patch deforms? Eg a football player’s jersey  Colors are “similar” but SSD won’t work Idea: patch histogram is stable To track:  repeat  predict location of new patch  search nearby for patch whose histogram matches original the best

16. When are motions easy? Current procedure  predict state  obtaining measurement from prediction by search  correct state Easy  When the object is close to where you expect it to be  eg Object guaranteed to move a little Large motions can be easy  When they’re “predictable”  e.g. ballistic motion  e.g. constant velocity Need a theory to fuse this procedure with motion model

17. General Model We assume there are moving objects, which have an underlying state X There are measurements Y, some of which are functions of this state Eg. There is a clock  at each tick, the state changes  at each tick, we get a new observation  object is ball, state is 3D position+velocity, measurements are stereo pairs  object is person, state is body configuration, measurements are frames, clock is in camera (30 fps)

18. Tracking as an Abstract Inference Problem Internal state: X Measurement: Y Major Steps: 1. Prediction 2. Data association: determining which data are informative 3. Correction

19. Independence Assumption Only the immediate past matters * X: Markov process Measurements depend only on the current state

20. Bayes Rule

21. Prediction

22. Correction

23. Linear Dynamic Model Model D: Transition matrix M: Measurement matrix Kalman filter is well-suited for this type of tracking Read Ch.11.3, Forsyth & Ponce, Computer Vision, 2012.

24. Kalman filter

25. Eg. Kalman filter *: predicted x: measured +: estimated o: true bar: 3 

26. Nonlinear Dynamics Problems  tend not to be normal  may not be Gaussian(quite common in vision problem)  multiple, well-separated modes

27. Eg. Nonlinear model

28. Eg. Nonlinear model Time Evolution & pdf

29. Difficulties in Complicated Models In order to maintain  handling multiple peaks  handling multi-dimensional state vector Particle filtering is a useful approach. How?

30. Sampled Representation Representation of pdf is  NOT for representing a pdf itself  BUT for computing some expectation Computing expectation using sampled representation

31. Monte Carlo Integration where : sampling distribution : weight

32. Obtaining a sampled representation of probability distribution

33. Computing an expectation using a set of samples

34. Transformation Transforming a sampled representation of a prior into that of a posterior Transforming a sampled representation of a prior into that of a posterior

35. Naive particle filter

36. Eg. Naive particle filter Poor results due to sample impoverishment!  Most of the weights get small very fast The way to get accurate estimates = to have samples lie where the p is likely to be large

37. Overcoming Sample Impoverishment Equivalent to maintaining a set of good particles Then, how? Resampling the prior 1. expand the sample set non-uniformly using the weights  Form a new set of samples consisting of a union of N k copies of (s k,1) for each k. 2. Subsample the sample set uniformly

38. Practical particle filter Initialization Prediction Correction Resampling:  Normalise the weights so that  Compute the variance of the normalised weights. If(var >Th) construct a new set of samples by drawing, with replacement, N samples from the old set, using the weights as the probability that a sample will be drawn. The weight of each sample is now 1/N. Naive particle filter

39. Consequence of resampling Particles that tend to reflect the state rather well usually reappear in the resampled set Many particles lie within one standard deviation of the mean of the posterior

40. Particle filters Different community -> different name statisticsparticle filter AIsurvival of the fittest computer visioncondensation

41. Tracking people Essential components  Motion model  Likelihood model P(image features|person present at given configuration) Motion model  Strong motion model: markers, angles,…  Weak motion model: drift model Likelihood model  SSD, edges, other features,…

42. Likelihood computation Boundary information Non-background information Sample points

43. Annealed particle filter To overcome the problems of  High dimensionality of the state  Many local peaks in the likelihood Annealing  Starts from smooth approximations to the likelihood to less smooth approximation  Repeats weighting and resampling [Deutscher,Blake,Reid, CVPR2000]

44. Finding people As an initialization of people tracker There is no person tracker that represents the configuration of the body and can start automatically Known approaches 1. Template matching 2. Finding faces 3. Search over correspondence * Challenging topic: learning models from data