1. Michal Irani Dept of Computer Science and Applied Math Weizmann Institute of Science Rehovot, ISRAEL Spatio-Temporal Analysis and Manipulation of Visual Information

2. 3. Geometric information 2. Extended temporal coverage Space-Time Visual Information VideoCollection of images 1. Extended spatial coverage

3. From Images to Space-Time Scenes Talk Outline: Single video sequence Multiple video sequences Multiple non-overlapping video sequences Efficient browsing, search, manipulation, enhancement, synthesis, editing, compression, and much much more... Space-time data redundancy RepresentationAnalysis Use it…

4. Video Browsing, Indexing, and Manipulation [work with P. Anandan, S. Hsu, T. Hassner] Redundant & implicit frame-based representation Compact & explicit scene-based representation 1. Extended spatial info 2. Extended temporal info 3. Geometric info

5. Video Browsing, Indexing, and Manipulation [work with P. Anandan, S. Hsu, T. Hassner] INTERACTIVE DEMO

6. Sequence 1 Sequence 2 Frame 1 Frame 2 Frame 3 Frame n Frame 1 Frame 2 Frame 3 Frame n Sequence-to-Sequence Alignment [work with Yaron Caspi - CVPR’00]

7. Sequence 1 Sequence 2 Frame 1 Frame 2 Frame 3 Frame n Frame 1 Frame 2 Frame 3 Frame n (a) Given corresponding frames. (b) Find spatial correspondences. (x,y) (x’,y’) Image-to-image alignment No dedicated hardware Should use all spatio-temporal information in video sequences ==> obtain correspondences in space and in time Image-to-Image Alignment Not enough spatial information

8. Not enough info for alignment in individual frames Image 1 Image 2 Information in Images:

9. Information in Video: Alignment uniquely defined Information cues for alignment: Appearance info Dynamic info within frames between frames Moving objects Non rigid motion Varying illumination

10. Spatio-Temporal Alignment SSD Minimization: Gauss-Newton (coarse-to-fine) iterations:

11. Sequence 1Sequence 2 Before AlignmentAfter Alignment

12. Sequence 1Sequence 2 Before AlignmentAfter Alignment

13. Sequence 1Sequence 2 Before AlignmentAfter Alignment Illumination changes:

14. Sequence 1Sequence 2 Before AlignmentAfter Alignment

15. Sequence 1Sequence 2 Before Alignment After Alignment

16. Alignment of Non-Overlapping Sequences Coherent appearance (Image-to-Image Alignment) Sequence-to-Sequence Alignment: Alignment in time and in space Coherent camera behavior Coherent scene dynamics (Seq-to-Seq Alignment) [work with Yaron Caspi - ICCV’01]

17. H=? Problem formulation H H Input: Output: and such that Sequence 1Sequence 2

18. Sequence 1: Sequence 2: Spatio-Temporal Alignment Combined Sequence:

19. Sequence 1: Sequence 2: Application: Wide-Screen Movies Wide- screen movie:

20. Fused Sequence: Visible light (video): Infra-Red: Application: Multi-Sensor Alignment

21. Zoomed-in Sequence: Zoomed-out Sequence: Application: Detect Zoomed Region

22. Wide Field-of-View Narrow Field-of-View Application: Spotting Hooligans

23. THE END Copyright, 1996 © Dale Carnegie & Associates, Inc. Summary Forget image frames Video >> collection of images Use all spatio-temporal info for representation, analysis, and exploitation.