CNN-based Action Recognition Using Adaptive Multiscale Depth Motion Maps And Stable Joint Distance Maps Junyou He, Hailun Xia, Chunyan Feng, Yunfei Chu Beijing University of Posts and Telecommunications GlobalSIP 2018 Nov. 27, 2018 2

Adaptive Multiscale Depth Motion Maps(AM-DMMs) OUTLINE Motivations The Proposed Method： Adaptive Multiscale Depth Motion Maps(AM-DMMs) Stable Joint Distance Maps(SJDMs) Input Preprocessing Network Training & Class Score Fusion Experiments Results Conclusions CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Advantages of depth modality: providing 3D structural information Motivations Advantages of depth modality: providing 3D structural information insensitive to variations in lighting Contains significant flicker noises But Depth map Skeleton data : more robust to noise But Not always reliable Each modality can capture a certain kind of information that is likely to be complementary to the other Thus, integrating the information from depth and skeleton is expected to improve the recognition performance CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The Spatio-Temporal Information Motivations Action Recognition The Spatio-Temporal Information Key Handcrafted features: SIFT, color histogram, edge direction … Domain knowledge Shallow & Dataset-dependent But Difficult to memorize the entire sequence information Difficult to extract high-level features But RNN-based methods Thus, we propose a compact and effective CNN based method to capture the spatio-temporal information . CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The Proposed Method Generate AM-DMMs Generate SJDMs Input Preprocessing Network Training & Class Score Fusion CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Adaptive Multiscale Depth Motion Maps(AM-DMMs) To capture more details of shape and motion information and cope with speed variations in actions Suffer from loss of temporal information DMMs AM-DMMs capture the detailed motion cues cope with speed variations CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Adaptive Multiscale Depth Motion Maps(AM-DMMs) AM-DMMs generated from a sample video of the action Swipeleft on three views DMM of a depth video sequence with N frames The motion energy E(i) of ith frame returns the number of non-zero elements in a binary map represents frame index is the projected map of Frame under projection view CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The Proposed Method Generate AM-DMMs Generate SJDMs Input Preprocessing Network Training & Class Score Fusion CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Stable Joint Distance Maps(SJDMs) To avoid excessive noise, three reference joints which are stable in most actions are used to compute relative distances of the other joints The Euclidean distance at frame t is the joint indices is one of three stable joints CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Stable Joint Distance Maps(SJDMs) The distances to different stable joints contain different spatial relationships and useful structural information of the skeleton is expressed as follows: corresponding to CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The Proposed Method Generate AM-DMMs Generate SJDMs Input Preprocessing Network Training & Class Score Fusion CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Input Preprocessing Resized maps to make them compatible with the pre-trained CNN model and solve the variable-length problem HSV-color coding has highlighted the differences in texture and edges Sample color coded AM-DMMs and SJDMs generated by the proposed method on UTD-MHAD dataset CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The Proposed Method Generate AM-DMMs Generate SJDMs Input Preprocessing Network Training & Class Score Fusion CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Network Training and Class Score Fusion A multi-channel CNN is adopted to exploit the discriminative features Two fusion methods are expressed as follows are score probability vectors is the element-wise multiplication are the accuracy of the corresponding network is a function to find the index of the element having the maximum score CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

skeleton joint positions inertial sensor signals Experiments: dataset UTD-MHAD dataset:multimodal action dataset RGB videos depth videos skeleton joint positions inertial sensor signals UTD-MHAD dataset: contains 27 different actions and each action is performed by 8 subjects (4 females and 4 males) and with up to 4 repetitions CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Experiments Result The effectiveness of different schemes and the results of individual CNN and two fusion methods Comparisons of the different scheme on UTDMHAD dataset CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Experiments Result The performance of the proposed method and the results reported before on UTD-MHAD dataset CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Experiments Result Confusion matrix of proposed method on the UTD-MHAD dataset CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

The proposed AM-DMMs capture more shape clues and details of motion. Conclusions Presents an effective method for action recognition using a nine-channel CNN The fusion of depth and skeleton modalities is proposed to improve the classification accuracy The proposed AM-DMMs capture more shape clues and details of motion. Transform one skeleton sequence into three SJDMs which describe different spatial relationships between joints CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Thanks! Junyou He @BUPT 12211006@bupt.edu.cn CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS