1. Self-Supervised Cross-View Action Synthesis
Kara Schatz
Advisor: Dr. Yogesh Rawat
UCF CRCV – REU, Summer 2019

2. Synthesize a video from an unseen view.
Project Goal
Synthesize a video from an unseen view.
Given:
video of the same scene from a different viewpoint
single image from the desired viewpoint

3. Last Week’s Results . . . . . . FRAME 1 FRAME 2 FRAME 15 FRAME 16
Ground
Truth:
. . .
Output:

4. Approach
Action
Rep.
Action
Rep.

5. Approach Action Rep. Action Rep.
This week, we went back to this original approach, but

6. Approach Action Rep. Action Rep.
This week, we went back to this original approach, but

7. Output without Consistency Loss
NTU: 66 epochs
Panoptic: 72 epochs
Input:
Output:
Ground Truth:
Input:
Output:
Ground Truth:
Frame 1:
Frame
16:

8. Output without Consistency Loss
NTU: 66 epochs
Panoptic: 72 epochs
Input:
Output:
Ground Truth:
Input:
Output:
Ground Truth:
Frame 1:
Frame
16:

9. Approach Trans- formation Trans-formation Estimated Key-points Action
viewpoint
Trans-
formation
Estimated
Keypoints
Key-points
Action
Rep.
Trans-formation
Estimated
Keypoints
Key-points
Action
Rep.
viewpoint

10. Output with Keypoints NTU: 98 epochs Panoptic: 108 epochs Input:
Ground Truth:
Input:
Output:
Ground Truth:
Frame 1:
Frame
16:

11. Output with Keypoints NTU: 98 epochs Panoptic: 108 epochs Input:
Ground Truth:
Input:
Output:
Ground Truth:
Frame 1:
Frame
16:

12. Keypoints
Keypoint
Transformed
View 1:
View 2:

13. Next Steps

14. Next Steps Trans- formation Trans-formation Estimated Key-points
viewpoint
Trans-
formation
Estimated
Keypoints
Key-points
Action
Rep.
Trans-formation
Estimated
Keypoints
Key-points
Action
Rep.
viewpoint

15. Next Steps Trans- formation Consistency losses Trans-formation
viewpoint
Trans-
formation
Estimated
Keypoints
Key-points
Action
Rep.
Consistency losses
Trans-formation
Estimated
Keypoints
Key-points
Action
Rep.
viewpoint

16. Next Steps Trans- formation Trans-formation Action Key-points
viewpoint
Trans-
formation
Action
Rep. Est.
Key-points
Action
Rep.
Action
Rep.
Trans-formation
Action
Rep. Est.
Key-points
viewpoint

17. Next Steps Predict Keypoints Trans- formation Predict Keypoints
viewpoint
Predict Keypoints
Trans-
formation
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
Predict Keypoints
Action
Rep.
Key-points
Predict Keypoints
Key-points
Action
Rep.
Trans-formation
Predict Keypoints
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
viewpoint

18. Next Steps Predict Keypoints Trans- formation Predict Keypoints
viewpoint
Predict Keypoints
Trans-
formation
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
Predict Keypoints
Action
Rep.
Key-points
Predict Keypoints
Key-points
Action
Rep.
Trans-formation
Predict Keypoints
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
viewpoint

19. Next Steps Predict Keypoints Trans- formation Predict Keypoints
viewpoint
Predict Keypoints
Trans-
formation
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
Predict Keypoints
Action
Rep.
Key-points
Predict Keypoints
Key-points
Action
Rep.
Trans-formation
Predict Keypoints
Estimated
Keypoints
Action
Rep. Est.
Key-points
Key-points
viewpoint