A Data-Driven Approach to Quantifying Natural Human Motion SIGGRAPH ’ 05 Liu Ren, Alton Patrick, Alexei A. Efros, Jassica K. Hodgins, and James M. Rehg Carnegie Mellon University & Georgia Institute of Technology Date: 8/24/2005 Speaker: Alvin

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion2 Outline Introduction Introduction Input Data Input Data Approaches Approaches Results Results Conclusions & Future Works Conclusions & Future Works

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion3 Introduction Goal Goal Quantify the naturalness of human motion Quantify the naturalness of human motion Solution Solution Train a classifier based on human-labeled data Train a classifier based on human-labeled data Train only on positive examples Train only on positive examples Assumption Assumption Motions that we have seen repeatedly are judged as natural Motions that we have seen repeatedly are judged as natural

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion4 Introduction cont. Application Application Verify the motion editing operations Verify the motion editing operations Contribution Contribution Pose the question Pose the question Decompose human motion into constituent parts Decompose human motion into constituent parts Contribute a substantial database Contribute a substantial database

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion5 Outline Introduction Introduction Input Data Input Data Approaches Approaches Results Results Conclusions & Future Works Conclusions & Future Works

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion6 Input Data Training Database Training Database Testing Motions Testing Motions

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion7 Training Database 1289 trials (422,413 frames) 1289 trials (422,413 frames) 34 subjects 34 subjects Vicon motion capture system of 12 cameras Vicon motion capture system of 12 cameras Downsample from 120Hz to 30 Hz Downsample from 120Hz to 30 Hz 41 markers 41 markers

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion8 Data Format ASF/AMC format ASF/AMC format Root position Root position Root orientation Root orientation Relative joint angles of 18 joints Relative joint angles of 18 joints 151-dimensional feature vector 151-dimensional feature vector Joint angle (4) and velocity (4) Joint angle (4) and velocity (4) Root ’ s linear velocity(3) and angular velocity(4) Root ’ s linear velocity(3) and angular velocity(4)

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion9 Testing Motions - Negative 170 trials, frames 170 trials, frames Edited motions Edited motions Keyframed motions Keyframed motions Noise Noise Motion Transitions Motion Transitions Insufficient cleaned motion capture data Insufficient cleaned motion capture data

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion10 Testing Motions - Positive 261 trials, frames 261 trials, frames MoCap MoCap Noise Noise Motion Transitions Motion Transitions Judge by an expert viewer Judge by an expert viewer

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion11 Outline Introduction Introduction Input Data Input Data Approaches Approaches Results Results Conclusions & Future Works Conclusions & Future Works

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion12 Approaches Framework Framework Mixture of Gaussians Mixture of Gaussians Hidden Markov Models Hidden Markov Models Switching Linear Dynamic System Switching Linear Dynamic System Naive Bayes (baseline method) Naive Bayes (baseline method) User Study User Study

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion13 Framework Select the statistical model Select the statistical model Fit the model parameters using natural human motions as training data Fit the model parameters using natural human motions as training data Compute a score for a novel input motion Compute a score for a novel input motion

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion14 Ensemble

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion15 Advantages of ensemble Avoid the problem of overfitting Avoid the problem of overfitting Detect the unnatural motions confined to a small set of joint angles Detect the unnatural motions confined to a small set of joint angles Provide guidance about what elements deserve the most attention Provide guidance about what elements deserve the most attention

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion16 Scoring

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion17 Mixture of Gaussians (MoG) The combinations of a finite number of Gaussian distributions Used to model complex multidimensional distributions EM algorithm is used to learn the parameters of the Gaussian mixture

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion18 MoG cont. 500 Gaussians 500 Gaussians Weak at modeling the dynamics Weak at modeling the dynamics

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion19 Hidden Markov Models (HMM)

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion20 HMM cont.

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion21 HMM cont. The distribution of poses is represented with a mixture of Gaussians The distribution of poses is represented with a mixture of Gaussians State was modeled as a single Gaussian State was modeled as a single Gaussian Parameters are learned by EM Parameters are learned by EM 180 hidden states for full body 180 hidden states for full body 60 hidden states for other feature group 60 hidden states for other feature group

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion22 Switching Linear Dynamic System (SLDS) State is associated with LDS instead of Gaussian distribution State is associated with LDS instead of Gaussian distribution Second-order auto-regressive (AR) model Second-order auto-regressive (AR) model Initial state is described by MoG Initial state is described by MoG Parameters are estimated using EM Parameters are estimated using EM 50 switching states for full body 50 switching states for full body 5 switching states for other feature group 5 switching states for other feature group

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion23 Principal Component Analysis HMM & SLDS HMM & SLDS 99% variance kept for the full-body model 99% variance kept for the full-body model 99.9% variance kept for the smaller model 99.9% variance kept for the smaller model

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion24 Naive Bayes (NB) Compute 1D marginal histogram for each feature over the entire training database Compute 1D marginal histogram for each feature over the entire training database Each histogram has 300 buckets Each histogram has 300 buckets Summing over the log likelihoods of each of the 151 features for each frame Summing over the log likelihoods of each of the 151 features for each frame Nomalizing the sum by the length Nomalizing the sum by the length

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion25 User Study 29 ♂ & 25 ♀ 29 ♂ & 25 ♀ 118 motion sequences 118 motion sequences 2 segments with a 10 minute break 2 segments with a 10 minute break The order of sequences is random The order of sequences is random

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion26 Outline Introduction Introduction Input Data Input Data Approaches Approaches Results Results Conclusions & Future Works Conclusions & Future Works

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion27 Results

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion28 Receiver Operating Characteristic Curve (ROC curve) False positive False positive Classifier predicts natural when the motion is unnatural Classifier predicts natural when the motion is unnatural True positive rate True positive rate tp / (tp + fn) tp / (tp + fn) False positive rate False positive rate fp / (fp + tn) fp / (fp + tn) Without need to choose a threshold Without need to choose a threshold The more area under the ROC curve, the more accurate the test The more area under the ROC curve, the more accurate the test

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion29

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion30

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion31

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion32

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion33 Demo

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion34 Outline Introduction Introduction Input Data Input Data Approaches Approaches Results Results Conclusions & Future Works Conclusions & Future Works

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion35 Conclusions Unusual motions are sometimes labeled unnatural (like falling) Unusual motions are sometimes labeled unnatural (like falling) Short errors and slow motions may not be detected Short errors and slow motions may not be detected Used to improve the performance of motion synthesis and motion editing tools Used to improve the performance of motion synthesis and motion editing tools

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion36 Future Works Explore dimensionality reduction approaches for SLDS model Explore dimensionality reduction approaches for SLDS model More sophisticated methods for normalizing or computing the score More sophisticated methods for normalizing or computing the score Screening for the style of a particular cartoon character Screening for the style of a particular cartoon character

Alvin\CAIG Lab\NCTU A Data-Driven Approach to Quantifying Natural Human Motion37 Thank you for your attention