Yao, B., and Fei-fei, L. IEEE Transactions on PAMI(2012) Recognizing Human-Object Interaction in still Image by Modeling the Mutual Context of Objects and Human Poses Date: 2013/05/27 Instructor: Prof. Wang, Sheng-Jyh Student: Hung, Fei-Fan Yao, B., and Fei-fei, L. IEEE Transactions on PAMI(2012)

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Why using context in computer vision? simple image vs. human activities Without context: With mutual context: ~3-4% with context without context

Challenges in Human Pose Estimation Human pose estimation is challenging  Object detection facilitate human pose estimation Difficult part appearance Self-occlusion Image region looks like a body part

Challenges in Object Detection Object detection is challenging human pose estimation facilitate object detection Small, low-resolution, partially occluded Image region similar to detection target

The Goal To build a mutual context model in Human-Object Interaction(HOI) activities To

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Model representation Modeling the mutual context of object and human poses A: Croquet shot Volleyball smash Tennis forehand Body parts Tennis ball Croquet mallet Volleyball Tennis racket O: Conditional random field without hidden variable 𝑂 1 , 𝑂 2 ,…, 𝑂 𝑀 , M:num of bounding box H: More than one atomic pose H in A P: body parts, 𝑃 1 , 𝑃 2 ,…, 𝑃 𝐿

Model representation 𝝓 𝟏 : co-occurrence compatibility between A,O,H activity H A P1 P2 PL O1 O2 Human pose objects 𝝓 𝟏 : co-occurrence compatibility between A,O,H 𝝓 𝟐 : spatial relationship between O,H 𝝓 𝟑 ~ 𝝓 𝟓 : modeling the image evidence with detectors or classifiers

𝝓1: Co-occurrence context co-occurrence between all A,O,H 𝜍 𝑖,𝑗,𝑘 : strength of co-occurrence interaction between ℎ 𝑖 , 𝑜 𝑗 , 𝑎 𝑘 H A P1 P2 PL O1 O2 𝟏 (∙) : indicator function 𝑁 ℎ : total number of atomic poses 𝑁 𝑜 : total number of objects 𝑁 𝑎 : total number of activity classes

𝝓2: Spatial context Spatial relationship between all O and different H 𝒙 𝐼 𝑙 : Spatial relationship between all O and different H 𝜆 𝑖,𝑗,𝑘 : weight of 𝑏 𝒙 𝐼 𝑙 , 𝑂 𝑚 𝑂 𝑚 = 𝑜 𝑗 𝑏 𝒙 𝐼 𝑙 , 𝑂 𝑚 : a sparse binary vector shows relative location of 𝑂 𝑚 w.r.t. 𝒙 𝐼 𝑙 H A P1 P2 PL O1 O2

𝝓3: Modeling objects Model O in the image I using object detection score For all object O 𝑔 𝑂 𝑚 : vector of score of detecting 𝑂 𝑚 𝛾 𝑗 : weight of 𝑔 𝑂 𝑚 𝑂 𝑚 = 𝑜 𝑗 Between Om and Om’ 𝑏 𝑂 𝑚 , 𝑂 𝑚′ : binary feature vector 𝛾 𝑗,𝑗′ : weight of 𝑜 𝑗 and 𝑜 𝑗′ H A P1 P2 PL O1 O2 𝑂 𝑚 =object in the mth bounding box 𝑏 𝑂 𝑚 , 𝑂 𝑚′ : binary feature vector show the spatial relationship between m & m’ 𝛾 𝑗,𝑗′ : weight for geometric configuration between 𝑜 𝑗 and 𝑜 𝑗′

𝝓4: Modeling human pose Model atomic pose that H belongs to and likelihood 𝑃(𝐼| ℎ 𝑖 ) 𝑃 𝒙 𝐼 𝑙 | 𝒙 ℎ 𝑖 𝑙 : Gaussian likelihood function 𝑓 𝑙 (𝐼) : vector of score of detecting body part in 𝒙 𝐼 𝑙 H A P1 P2 PL O1 O2 𝑃 𝒙 𝐼 𝑙 | 𝒙 ℎ 𝑖 𝑙 : Gaussian likelihood of body part given atomic pose

𝝓5: Modeling activity Model HOI activity by training activity classifier 𝑠 𝐼 : 𝑁 𝑎 -dim output of one-versus-all (OVA) discriminative classifier taking image as features 𝜂 𝑘 : feature weight of 𝑎 𝑘 H A P1 P2 PL O1 O2

One-versus-all classifier OVA: OVO:

Model Properties Spatial context between O and H Object detection and human pose estimation facilitate each other Ignore the objects and body parts that are unreliable Flexible to extend to large scale datasets and other activities Jointly model can share all objects and atomic poses

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Training detectors and classifiers Model Learning Assign human pose to atomic pose Training detectors and classifiers Estimate parameters by Maximum Likelihood

Obtaining Atomic Poses Assign human pose to atomic pose Using clustering to obtain atomic poses Normalize the annotations 𝒙 1 , 𝒙 2 ,…, 𝒙 𝐿 Finding missing part Using the nearest visible neighbor Obtain a set of atomic poses Hierarchical clustering with maximum linkage measure : 𝑙=1 𝐿 𝑤 𝑇 | 𝒙 𝑖 𝑙 − 𝒙 𝑗 𝑙 | Training detectors and classifiers Estimate parameters by Maximum Likelihood

Training Detectors and Classifiers Assign human pose to atomic pose 𝑔 𝑂 𝑚 : Object detector in 𝜙 3 𝑂,𝐼 𝑓 𝑙 (𝐼) : Human body part detector in 𝜙 4 𝐻,𝐼 𝑠 𝐼 : Overall activity classifier in 𝜙 5 (𝐴,𝐼)  deformable part model Training detectors and classifiers Spatial pyramid matching (SPM) SIFT + 3 level image pyramid Estimate parameters by Maximum Likelihood

Spatial pyramid matching SIFT 3-level histogram intersection kernel S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006.

Spatial pyramid matching

Estimating Model Parameters Assign human pose to atomic pose Estimate 𝜍, 𝜆, 𝛾, 𝛼, 𝛽 by using ML approach with zero-mean Gaussian prior Training detectors and classifiers Estimate parameters by Maximum Likelihood

Learning result

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Update object detection results Model Inference New image Update human body parts Update object detection results Initialize with learned results Update A and H labels

Activity classification Initialization New image A: SPM classification O: object detection H: pictorial structure model Initialize with learned results Initialize Activity classification Object detection Human pose estimation

Update model inference Marginal distribution of human pose: 𝑝(𝐻= ℎ 𝑖 ) 𝑖=1 𝑁 ℎ Using mixture of Gaussian to refine the prior of body part 𝒩( 𝒙 ℎ 𝑖 𝑙 ) 𝑖=1 𝑁 ℎ 𝑝(𝐻= ℎ 𝑖 ) 𝒩( 𝒙 ℎ 𝑖 𝑙 ) Update human body parts Update object detection results Marginal distribution of human pose 看哪一個 human pose H 的機率最高 Update A and H labels

Update model inference O,H O,A,H O,I Update human body parts Update object detection results Greedy forward search method : Initial (𝑚,𝑗) and no object in bounding box Select 𝑚 ∗ , 𝑗 ∗ =𝑎𝑟𝑔𝑚𝑎𝑥 (𝑚,𝑗) Label 𝑚 ∗ box as 𝑜 𝑗 ∗ update (𝑚,𝑗) Stop when 𝑚 ∗ , 𝑗 ∗ <0 Score of the mth object bounding box to object oj Update A and H labels

Update model inference Enumerate possible A and H label Optimize Ψ(𝐴,𝑂,𝐻,𝐼) Update human body parts Update object detection results Based on results of Human pose estimation and Object detection Update A and H labels

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Experimental Results (Sports Dataset)

Experimental Results (Sports Dataset)

Experimental Results (Sports Dataset) Activity classification

Experimental results (PPMI Dataset)

Experimental results (PPMI Dataset)

Outline Introduction Model Representation Model Learning Intuition and goal Model Representation Model Learning Obtaining Atomic Poses Training Detectors and Classifiers Estimating Model Parameters Model Inference Experimental Results Conclusion

Conclusion Mutual context can significantly improve the performance in difficult visual recognition problems The joint model can share all the information Annotate all the human body parts and objects in training images

Reference Yao, B., and Fei-fei, L. “Recognizing Human-Object Interactions in Still Images by Modeling the Mutual Context of Objects and Human Poses,” IEEE Transactions on Pattern Analysis and Machine Intelligence (2012) B. Yao and L. Fei-Fei, “Modeling Mutual Context of Object and Human Pose in Human-Object Interaction Activities,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010 B. Sapp, A. Toshev, and B. Taskar, “Cascade Models for Articulated Pose Estimation,” Proc. European Conf. Computer Vision, 2010. S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006. http://en.wikipedia.org/wiki/Hierarchical_clustering

H A P1 P2 PL O1 O2 H A P1 P2 PL O1 O2