1. Appearance Transformer (AT)
Cross-View Action Synthesis Kara Schatz, Dr. Yogesh Rawat, Dr. Mubarak Shah
Goal and Motivation
Approach
Results
Synthesize action videos from an unseeen view
Input: video sequence from different view
single image of desired view
Humans can perform this task easily, can machines?
Consistency loss between views
Keypoint detection
Transformation of video and appearance features
Using NTU dataset with randomly chosen views
Without Appearance Transformer and Keypoints – 400 epochs
Input
Ground
Truth
Network Architecture
Output
View 1
App. View 1
Trans.
App. 1
Appearance
Transformer
Reconstruction
Loss (MSE)
VGG
Input
Video View 2
Est. Rep.
View 1
Generator
Video View 1
Ground
Truth
Rep.
View 1
I3D
Est. KP
View 1
Core
Network
Output
Video View 2
Rep.
View 2
I3D
Est. KP
View 2
With Appearance Transformation, no Keypoints – 109 epochs
Video View 2
Est. Rep.
View 2
Generator
Input
View 2
Reconstruction
Loss (MSE)
App. View 2
Trans.
App. 2
Appearance
Transformer
VGG
Ground
Truth
Output
Core Network
Appearance Transformer (AT)
Rep.
View 1
Input
Keypoint Predictor KP
View 1
Est. Rep.
timestep 1
Est. Rep.
timestep 2
Est. Rep.
timestep n
Consistency
Loss (MSE)
Consistency
Loss (MSE)
Ground
Truth
viewpoint
change
Trans-
former
Est. Rep.
View 1
Keypoint Predictor
Est. KP
View 1
Appear-ance AT
Cell AT
Cell
. . . AT
Cell
Output
viewpoint
change
Trans-former
Est. Rep.
View 2
Keypoint Predictor
Est. KP
View 2
Future Work
Consistency
Loss (MSE)
Consistency
Loss (MSE)
Trans. App.
timestep 1
Trans. App.
timestep 2
Trans. App.
timestep n
Improve appearance transformation module
Use pixel-weighted loss
Use attention
Rep.
View 2
Keypoint Predictor KP
View 2