1. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Heterogeneous Sensor Webs for Automated Target Recognition and Tracking in Urban Terrain (W911NF-06-1-0076) Motion Pattern Analysis with(out) Trajectories John Fisher MIT CSAIL

2. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with people What was he thinking? Hard questions, most 9- year olds wouldn’t know the answer anyway. What was he reacting to? Perhaps a little easier. Can we analyze the dependency between the player interactions?

3. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with people Two camera angles provide noisy position estimates. Can derive noisy estimates of kinematic state (i.e. position, velocity, etc.) Could use other features as well (e.g. body position, etc.) This gives multiple time series. One for each player plus the ball. Interaction can be cast as inference over the structure of influence between time-series.

4. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with their environments Object tracking is challenging in these scenarios. Can we derive aggregate models of behavior? Assume there is a persistent or slowly varying motion pattern. Can we estimate aggregate motion patterns without tracking each individual?

5. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Motion Pattern Analysis Inference over aggregate properties of a dynamic scene Scenarios in which tracking each object is intractable – Challenging when the properties live in a curved space Integration of two mathematical formalisms – Lie-algebraic representations of motion/deformation – Variational inference in graphical models Inference over the structure of interactions between multiple time- series Suppose we are only interested in the graph describing interactions – Complexity of inference over structure is super-exponential O(N N ) in the number of objects – If the structure of interaction varies dynamically, complexity is exponential in the duration O((N N ) T )

6. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Lie Groups and Lie Algebras Connected by matrix exponentiation and logarithm Other properties – Identity transform corresponds to zero vector – Inverse transforms corresponds to negation – Commutable multiplication corresponds to addition

7. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Acting on geometric points Acting on images Lie Group Action on Images Lie group action Two roles of the transform T Twofold role is key for estimating T directly from images

8. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Action of the infinitesimal generator T: transform, X: infinitesimal generator

9. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 The key relation We derive the following relation between the motion space and the space of image changes the decomposition of motion the decomposition of image changes The motion can be inferred by decomposing the image changes

10. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Computation Assume that the motion is characterized by approximate by finite difference negated pointwise inner product of the gradient and the induced velocity

11. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Extension to Triangular Meshes Extend the formalism with triangle mesh Consistency at boundary Consistent subspace (m triangles, n vertices) joint dim = -) constraints = subs. dim = The key relation still applies with this extension.

12. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Efficient Inference Over Deformations -  finite difference p.w. dot product …… gradient estimate by regression

13. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Multi-flow probabilistic model M M models flow indicator flow indicator MRF

14. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with their environments (Lie group)

15. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with their environments (Lie group)

16. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 People interacting with their environments (opt flow)

17. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Object Interaction Analysis N tracked objects (cars, people, genes, consumers, etc..) Observe T noisy samples of state (e.g. position/velocity) Can we infer properties of the interaction? Who is interacting with whom? Who is the probable leader? Is the nature of interaction changing over time?

18. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Outline Challenges of Dependence Analysis – Static Dependence – Dynamic Dependence Factorization Model Temporal Interaction Model Conclusion 18

19. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Static Dependence Analysis Given – Observations: – Generative Model: – Prior: Find 19

20. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Challenges of Static Analysis Structural specification Unknown parameters Number of structures – e.g. We explore > 109 structures 20

21. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Temporal Interaction Model 1 2 3

22. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Conjugate Prior on Parameters Given the structure, parameters are independent: – and modular: is the same for all such that 22 1 2 3 1 2 3

23. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Conjugate Prior on Structures The prior on structure factorizes as a product of weights on parent sets: 23 UniformDenseSparse

24. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Posterior Posterior on structure is a simple update to the parent set weights where: 24 Matrix-T Matrix Normal-Inverse-Wishart Matrix Normal

25. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Computing the Partition Function How many structures are there? – Each time-series has possible parents – time-series 25 Super-exponential in N parent sets structures

26. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Computing the Partition Function 26 All Directed Structures Super-exponential to Exponential

27. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Bounded Parent Sets Assume, – structures (still super-exponential) 27 Polynomial-time computation

28. MURI Annual Review, Vanderbilt, Sep 8 th, 2009"Motion Pattern Analysis", Fisher28 Switching Vector Autoregressive Tree SVART(1)

29. MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Bayesian Reasoning over Interaction Structures

30. MURI Annual Review, Vanderbilt, Sep 8 th, 2009"Motion Pattern Analysis", Fisher30 Basketball

31. MURI Annual Review, Vanderbilt, Sep 8 th, 200931 Basketball Results Using a STIM with 10 states with Team A on Offense Team B on Offense Team B trans. to Offense Team A trans. to Offense

32. MURI Annual Review, Vanderbilt, Sep 8 th, 200932 Basketball Event Probabilities Team A on Offense Team B on Offense Team B trans. to Offense Team A trans. to Offense Team A on Offense Team B on Offense Team B trans. to Offense Team A trans. to Offense

33. MURI Annual Review, Vanderbilt, Sep 8 th, 200933 Expected Number of Children Top 4 – Ball (1.8) – Point Guard A (1.7) – Forward 1 A (1.1) – Forward 1 B (1.0)

34. MURI Annual Review, Vanderbilt, Sep 8 th, 200934 Influence of Point Guard A 0.01.00.5

35. MURI Annual Review, Vanderbilt, Sep 8 th, 200917Apr08 Page 35 Prior and Posterior Expectations Can tractably compute expectations of: Multiplicative functions on structure Additive functions on structure Allows one to calculate: Expected number of children – How influential? Expected number of parents – How impressionable?

36. MURI Annual Review, Vanderbilt, Sep 8 th, 200917Apr08 Page 36 Comments Focus was on vision sensors, but both methods have wider application – Seismic data (petroleum exploration) – Audio-visual association (multi-media annotation) Thank you