Sequence-to-Sequence Alignment and Applications. Video > Collection of image frames.

Slides:

Advertisements

Similar presentations

An Idiot’s Guide to Exposure a.k.a. John’s Guide to Exposure.

Advertisements

ISO, Aperture and Shutter Speed For Beginners. The photographer can control how much natural light reaches the sensor by adjusting the camera's ISO shutter.

High-Resolution Three- Dimensional Sensing of Fast Deforming Objects Philip Fong Florian Buron Stanford University This work supported by:

www-video.eecs.berkeley.edu/research

Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Where does the.

Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.

Space-time interest points Computational Vision and Active Perception Laboratory (CVAP) Dept of Numerical Analysis and Computer Science KTH (Royal Institute.

EVENTS: INRIA Work Review Nov 18 th, Madrid.

Light Field Rendering Shijin Kong Lijie Heng.

December 5, 2013Computer Vision Lecture 20: Hidden Markov Models/Depth 1 Stereo Vision Due to the limited resolution of images, increasing the baseline.

Local Descriptors for Spatio-Temporal Recognition

X From Video - Seminar By Randa Khayr Eli Shechtman, Yaron Caspi & Michal Irani.

Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.

Geometry of Images Pinhole camera, projection A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix.

Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.

Image-Based Rendering Produce a new image from real images. Combining images Interpolation More exotic methods.

Modeling the imaging system Why? If a customer gives you specification of what they wish to see, in what environment the system should perform, you as.

CCU VISION LABORATORY Object Speed Measurements Using Motion Blurred Images 林惠勇中正大學電機系

Image Stitching and Panoramas

CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai.

Combining Laser Scans Yong Joo Kil 1, Boris Mederos 2, and Nina Amenta 1 1 Department of Computer Science, University of California at Davis 2 Instituto.

Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure This lesson – unconstrained flow-fields Next lesson.

Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.

Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.

CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.

CIS 681 Distributed Ray Tracing. CIS 681 Anti-Aliasing Graphics as signal processing –Scene description: continuous signal –Sample –digital representation.

Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.

CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai.

3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.

A plane-plus-parallax algorithm Basic Model: When FOV is not very large and the camera motion has a small rotation, the 2D displacement (u,v) of an image.

Super-Resolution Dr. Yossi Rubner

Mohammed Rizwan Adil, Chidambaram Alagappan., and Swathi Dumpala Basaveswara.

Goals For This Class Quickly review of the main results from last class Convolution and Cross-correlation Discrete Fourier Analysis: Important Considerations.

Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5409 T-R 10:30am – 11:50am.

Camera Calibration & Stereo Reconstruction Jinxiang Chai.

Action recognition with improved trajectories

Jitter Camera: High Resolution Video from a Low Resolution Detector Moshe Ben-Ezra, Assaf Zomet and Shree K. Nayar IEEE CVPR Conference June 2004, Washington.

IRISA / INRIA Rennes Computational Vision and Active Perception Laboratory (CVAP) KTH (Royal Institute of Technology)

CAP5415: Computer Vision Lecture 4: Image Pyramids, Image Statistics, Denoising Fall 2006.

The Brightness Constraint

Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney Princeton University COS 429 Lecture Feb. 12, 2004.

The Measurement of Visual Motion P. Anandan Microsoft Research.

December 4, 2014Computer Vision Lecture 22: Depth 1 Stereo Vision Comparing the similar triangles PMC l and p l LC l, we get: Similarly, for PNC r and.

Yu-Wing Tai, Hao Du, Michael S. Brown, Stephen Lin CVPR’08 (Longer Version in Revision at IEEE Trans PAMI) Google Search: Video Deblurring Spatially Varying.

Stereo Many slides adapted from Steve Seitz.

Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA.

Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Amit Agrawal and Ramesh Raskar Mitsubishi Electric Research.

Recognizing Action at a Distance Alexei A. Efros, Alexander C. Berg, Greg Mori, Jitendra Malik Computer Science Division, UC Berkeley Presented by Pundik.

Uses of Motion 3D shape reconstruction Segment objects based on motion cues Recognize events and activities Improve video quality Track objects Correct.

Image Processing Basics. What are images? An image is a 2-d rectilinear array of pixels.

Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm Lecture #16.

0 Assignment 1 (Due: 3/9) The projections of two parallel lines, l 1 and l 2, which lie on the ground plane G, onto the image plane I converge at a point.

R&D  BBC MMX Broadcast-related challenges: Increasing quality and interactivity of audio-visual media Graham Thomas BBC R&D

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman.

Photo-realistic Rendering and Global Illumination in Computer Graphics Spring 2012 Hybrid Algorithms K. H. Ko School of Mechatronics Gwangju Institute.

Motion Estimation I What affects the induced image motion?

infinity-project.org Engineering education for today’s classroom Outline Images Then and Now Digitizing Images Design Choices in Digital Images Better.

Jianchao Yang, John Wright, Thomas Huang, Yi Ma CVPR 2008 Image Super-Resolution as Sparse Representation of Raw Image Patches.

Michal Irani Dept of Computer Science and Applied Math Weizmann Institute of Science Rehovot, ISRAEL Spatio-Temporal Analysis and Manipulation of Visual.

Compressive Coded Aperture Video Reconstruction

Image Restoration using Model-based Tracking

Computer Vision, Robotics, Machine Learning and Control Lab

Distributed Ray Tracing

Optical Flow Estimation and Segmentation of Moving Dynamic Textures

Presented by Omer Shakil

Range Imaging Through Triangulation

Outline Linear Shift-invariant system Linear filters

Lecture 10 Causal Estimation of 3D Structure and Motion

A guide to SR different approaches

Filtering Things to take away from this lecture An image as a function

Presentation transcript:

Sequence-to-Sequence Alignment and Applications

Video > Collection of image frames

= Space-time volume X Y Time

Sequence-to-Sequence Alignment [work with Yaron Caspi] Sequence 1 Frame 1 Frame 2 Frame 3 Frame n Sequence 2 Frame 1 Frame 2 Frame 3 Frame n

Video 2 Video 1 Frame 1 Frame 2 Frame 3 Frame n Frame 1 Frame 2 Frame 3 Frame n (a) Find temporal correspondences (b) Find spatial correspondences (x,y,t) (x’,y’,t’) x y t Align and Integrate Space-Time Info [work with Yaron Caspi]

Spatial resolution Temporal resolution Spectral range Depth of focus Dynamic range Field-of-View (FOV) View point “Super Sensors” Exceed Optical Bounds of Visual Sensors: Align and Integrate space-time info

Not enough info for alignment in individual frames Image 1 Image 2 Image-to-Image Alignment

Information in Video: Alignment uniquely defined Appearance info Dynamic info within frames between frames Moving objects Non rigid motion Varying illumination

Where: Problem Formulation

Spatio-Temporal Alignment SSD Minimization: Gauss-Newton (coarse-to-fine) iterations

Coarse-to-Fine Minimization time Sequence 1 time Sequence 2 Pyramid of Sequence 2 Pyramid of Sequence … …

Sequence 1Sequence 2 Before AlignmentAfter Alignment

Sequence 1Sequence 2 Before AlignmentAfter Alignment

Sequence 1Sequence 2 Before AlignmentAfter Alignment

Sequence 1Sequence 2 Before AlignmentAfter Alignment Illumination changes:

Sequence 1Sequence 2 Before Alignment After Alignment

time Super-resolution in space and in time. time High-resolution output sequence: time Low-resolution input sequences Increasing Space-Time Resolution in Video [work with Eli Shechtman & Yaron Caspi]

Spatial Super-Resolution Multiple low-resolution input images: High-resolution output image: Recover small details

What is Super-Resolution in Time? Recover dynamic events that are “faster” than frame-rate (Generate a “high-speed” camera) Application areas: sports events, scientific imaging, etc... Effects of “fast” events imaged by “slow cameras”: (1) Motion aliasing (2) Motion blur

(1) Motion Aliasing The “Wagon wheel” effect:Slow-motion: time Continuous signal time Sub-sampled in time time “Slow motion”

(2) Motion Blur

S h (x h,y h,t h ) Space-Time Super-Resolution x y t y x t Blur kernel: PSF Exposure time Low resolution input sequences High-resolution space-time volume

Super Resolution in Time Input 1Input 2 Input 3Input 4 (25 frames/sec)

Input sequence in slow motion: Super Resolution in Time Output sequence (super-resolved) : (75 frames/sec)

Motion Blur

Overlay of frames Simulated sequences of “fast” event: Very long exposure-time Very low frame-rate One low-res sequence: Another low-res sequence:And another one... Motion Blur

Output trajectory: (overlay of frames) Deblurring: 3 out of 18 low-resolution input sequences: (frame overlays) Output: Input: Output sequence: (x15 frame-rate) Without estimating motion of the ball!

Input (low-res) frames at collision: 4 input sequences: Output (high-res) frame at collision: Motion-Blur Video 1 Video 3 Video 2 Video 4

Spatial resolution Temporal resolution Spectral range Depth of focus Dynamic range Field-of-View (FOV) View point Optical Limits of Visual Sensors: Very little common visual information!!!

Alignment of Non-Overlapping Sequences Coherent appearance (Image-to-Image Alignment) Sequence-to-Sequence Alignment: Alignment in time and in space Coherent camera behavior Coherent scene dynamics (Seq-to-Seq Alignment) [work with Yaron Caspi]

The scene When is it possible? 2) cameras fixed relative to each other 1) same center of projection

H=? Problem formulation H H Input: Output: and such that Sequence 1Sequence 2 Conjugate matrices have the same eigenvalues:

Recovering Temporal Alignment =?  T and S have the same eigenvalues, up to scale: Search for the temporal shift which minimizes:

Recovering Spatial Transformation Given : Solve a homogeneous set of linear equations in H

Sequence 1: Sequence 2: Exceed Limited FOV Combined Sequence:

Sequence 1: Sequence 2: Exceed Limited Field of View – Wide-Screen Movies Wide- screen movie:

Fused Sequence: Visible light (video): Infra-Red: Exceed Limited Spectral Range – Day and Night Vision

Zoomed-outZoomed-in Exceed Limited Focal Length –

Zoomed-in Zoomed-out Exceed Limited Focal Length

Copyright, 1996 © Dale Carnegie & Associates, Inc. Summary Forget image frames Video = space-time volume >> collection of images Use all available spatio-temporal info for analysis, representation, and exploitation. Applies to many problem areas: 1. Quick search in video. 2. Alignment and integration of information to exceed optical bounds of visual sensors. 3. Action analysis and recognition 4. Synthesis of video data and many more…

Copyright, 1996 © Dale Carnegie & Associates, Inc. A few comments and clarifications regarding Exercise 4 ON THE BOARD (Please ask a friend if you were not in class)