1. KOUROSH MESHGI PROGRESS REPORT TOPIC To: Ishii Lab Members, Dr. Shin-ichi Maeda, Dr. Shigeuki Oba, And Prof. Shin Ishii 9 MAY 2014

2. KOUROSH MESHGI – ISHII LAB - DEC 2013 - SLIDE 2 MAIN APPLICATIONS Surveillance Public Entertainment Robotics Video Indexing Action Recog.

3. KOUROSH MESHGI – ISHII LAB - DEC 2013 - SLIDE 3 MAIN CHALLENGES Varying Scale Clutter Deformation Illumination Abrupt Motion

4.  Goal: Define p(X t |Y 1,…,Y t ) given p(X 1 ) KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 4 States: Target Location and Scale Observations: Sensory Information

5. PARTICLE FILTER TR.

6. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 6 Frame: t RGB Domain

7. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 7 Frame: t Depth Domain

8. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 8 Frame: t Sensory Information

9. Observation KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 9 Frame: t State w h (x,y)

10. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 10 Feature Set Color Shape Edge Texture

11. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 11 Frame: 1 Template f1f1 fjfj fMfM ……

12. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 12 Frame: 1 Particles Initialized Overlapped

13. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 13 Frame: t Motion Model → t + 1

14. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 14 Frame: t + 1 Feature Vectors f1f1 f2f2 fMfM X 1,t+1 X 2,t+1 X N,t+1 … …

15. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 15 Frame: t Probability of Observation Each Feature Independence Assumption  !

16. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 16 Frame: t + 1 Particles Brighter = More Probable

17. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 17 Frame: t + 1 Feature Vectors

18. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 18 Frame: t + 1 New Model Model Update

19. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 19 Frame: t + 1 Proportional to Probability

20. PARTICLE FILTER TR.

21. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 21 Appearance ChangesModel DriftDeficient Feature SpaceUninformed SearchOptimized Feature SelectionApproximation of Target

22. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 22 Same Color Objects Background Clutter Illumination Change Shadows, Shades Use Depth!

23. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 23 Templates Corrupted! Handle Occlusion! (No Model Update During Them)

24. * Local Optima of Feature Space * Feature Noise * Feature Failures KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 24 Regularization Non-zero Values Normalization

25. Particles Converge to Local Optima / Remains The Same Region KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 25 Advanced Motion Models (not always feasible) Restart Tracking (slow occlusion recovery) Expand Search Area!

26. * The Search is not Directed * Neither of the Channels have Useful Information * Particles Should Scatter Away from Last Known Position KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 26 Occlusion!

27. do not address occlusion explicitly maintain a large set of hypotheses  computationally expensive direct occlusion detection robust against partial & temp occ.  persistent occ. hinder tracking GENERATIVE MODELS DISCRIMINATIVE MODELS Dynamic Occlusion: Pixels of other object close to camera Scene Occlusion: Still objects are closer to camera than the target object Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ UPDATE MODEL FOR TARGET  TYPE OF OCCLUSION IS IMPORTANT  KEEP MEMORY VS. KEEP FOCUS ON THE TARGET Combine Them!

28.  PTO partial occlusion  SAO self- or articulation occlusion  TFO temporal full occlusion - shorter than 3 frames  PFO persistent full occlusion  CPO complex partial occlusion - including “split and merge” and permanent changes in a key attribute of a part of target  CFO complex full occlusion KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 28

29. [Zhao & Nevatia, 04] Occlusion Indicator: Ratio of FG/BKG [Wu & Nevatia, 07] Handle Occlusion using Appearance Model [de Villiers et al., 12] Switch Tracker in the case of Occlusion [Song & Xiao, 13] Occlusion Indicator: New Peak in HOD or Reduction of the Size of Main Peak Many other papers handle occlusions as the by- product of their novel trackers

30. OCCLUSION AWARE PFT

31. Motion Model Resampling Target Estimation Calculate Likelihood KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 31 Initialization Model Update Observation Occlusion Flag? Constant Likelihood Occlusion Estimation Occlusion Threshold > ? YES NO

32.  Occlusion Flag (for each particle)  Observation Model  No-Occlusion Particles  Same as Before  Occlusion-Flagged Particles  Uniform Distribution KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 32

33.  Position Estimation of the Target  Occlusion State for the Next Box KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 33 1 0 1 0 a x x

34.  Model Update (Separately for each Feature)  Modified Dynamics Model of Particle KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 34

35. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 35 Occlusion!

36. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 36 Occlusion! GOTCHA!

37. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 37 Quick Occlusion Recovery  Low CPE No Template Corruption No Attraction to other Object/ Background

38. COLOR (HOC) TEXTURE (LBP) EDGE (LOG) 2D PROJ. (BETA) 3D SHAPE (PCL Σ ) DEPTH (HOD) GRADIENT (HOG) KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 38

39. & DISCUSSION

40. Princeton Tracking Dataset 5 Validation Video with Ground Truth 95 Evaluation Video KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 40

41. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 41 OAPFT (Proposed, with different feature sets) State-of-the-art RGBD tracker OI+SVM (SVM tracker with Occlusion Indicator) Traditional Particle Filter tracker ACPF (Adaptive Color Particle Filter) State-of-the-art RGB tracker, Successful for Occlusion Handling STRUCK (Structured Output SVM Tracker)

42.  PASCAL VOC: Overall Performance toto Success Overlap Threshold 0 1 1 Area Under Curve KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 42

43. KOUROSH MESHGI – ISHII LAB - MAR 2014- SLIDE 43 1 1 Success Plot Overlap Threshold Success Rate 1 1

44.  Mean Central Point Error: Localization Success  Mean Scale Adaption Error KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 44 EstimatedGround Truth

45. Center Positioning Error 400 50 Frames CPE (pixels)

46. Scale Adaptation Error 140 50 Frames SAE (pixels)

47.  FP happens when a tracker doesn’t realize that the target is occluded.  MI happens when the target is visible but the tracker fails to track it as if the target is still in an occlusion state  MT the estimated bounding box has nothing in common with ground truth box  FPS execution time in frames per second KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 47

48. KOUROSH MESHGI – ISHII LAB – MAR 2014 – SLIDE 48 Tracker AUCCPESAEMIFPMTFPS BCDEGST (proposed) 76.509.597.320.02.40.00.9 ACPF (Nummiaro ‘03) 27.5590.3835.2712.60.031.01.4 STRUCK (Hare ‘11) 46.6768.7426.6112.60.064.413.4 OI+SVM (Song ‘13) 69.159.6812.040.420.00.80.4

49. KOUROSH MESHGI – ISHII LAB – MAY 2014 – SLIDE 49 More Resilient Features + Scale Adaptation Active Occlusion Handling Measure the Confidence of each Data Channel Adaptive Model Update

50. Q UESTIONS? Thank you for your time…