X From Video - Seminar By Randa Khayr Eli Shechtman, Yaron Caspi & Michal Irani

Space-Time Super Resolution is the increase of temporal and spatial resolution in video, using multiple video sequences describing the same event. multiple video sequences describing the same event. Rapid movements and high spatial frequencies, which are otherwise lost, are thus recovered. This algorithm requires no a-priori knowledge of the movement inside the frame. Investigate the algorithm's capabilities and limitations. Several examples of using the algorithm are shown.

Limited Temporal resolution resolution Spatial resolution

The spatial resolution is determent by : spatial density of the detectors in camera. The temporal resolution is determent by: the frame-rate and by exposure time of the camera.

This problem is often evident in sports videos. There are two typical visual effects in video sequences which are caused by very fast motion. sequences which are caused by very fast motion.

Is caused by the exposure time of the camera. Fig. 1. Motion blur. Distorted shape due to motion blur of very fast moving objects (the tennis ball and the racket) in a real tennis video. The perceived distortion of the ball is marked by a white arrow. Note, the “V”- like shape of the ball in (a), and the elongated shape of the ball in (b). The racket has almost “disappeared”.

is due to the temporal sub-sampling introduced by the frame-rate of the camera. Fig. 2. Motion aliasing. (a) shows a ball moving in a sinusoidal trajectory. (b) displays an image sequence of the ball captured at low frame-rate. The perceived motion is along a straight line. This false perception is referred to in the paper as “motion aliasing”. (c) Illustrates that even using an ideal temporal interpolation will not produces the correct motion. The filled-in frames are indicated by the blue dashed line.

“wagon wheel effect”:

Wrong answer

Spatial and temporal dimensions are : Different but inter-related. Combine input sequences of different ST resolutions. Combine information from Better video

S : a dynamic space-time scene. : n video sequences of that scene recorded by n different video cameras. Have limited spatial and temporal resolution,due to the : Space-time imaging process, Blurring Sampling in time and Space

Spatial sampling Finite number of detectors Temporal sampling Finite frame-rate Video of low space-time resolution Use information from that sequences to construct new sequence of high space-time resolution PAL:25 frames/sec NTSC:30 frames/sec

Different possible discretizations of the space-time volume resulting in two different high resolution output sequences.

In general a space-time scene is capture by a 4D representation (x,y,z,t). We will deal with dynamic scenes which can be modeled by a 3D space-time volume (x,y,t). The scene is planner, and the dynamic events occur within this plane. The scene is general dynamic 3D scene, but the distances Between the recording video cameras are small relative To there distance from the scene.

“ reference ” sequence whose axes are aligned with those of the continuous space-time volume S S : The unknown dynamic scene we wish to reconstruct :Is a discretization of S with a higher sampling rate than that of : Scaling transformation :Transformation :Relation,

Modeled by a 1-D affine transformation in time Temporal misalignment Spatial misalignment Modeled by an inter-camera homography Our space-time super-resolution algorithm does not require knowledge of these motions, only the knowledge of It can thus handle very complex dynamic scenes.

The Space-Time Imaging Model Denote the combined space-time blur operator of the ith camera corresponding to the low resolution space-time point The corresponding high resolution space-time point Is a point-dependent space-time Blur kernel represented in the high resolution coordinate system.

Then the relation between the unknown space-time values and the known low resolution space-time measurements can be expressed by: (For video cameras with different photometric responses we just add preprocessing step)

Temporal Super-Resolution (a) (b)(c) (d) (e) (f)

Low temporal-resolution input sequence Super-Resolved Output

The “ wagon wheel effect ”

Space-Time Visual Tradeoffs The spatial and temporal dimensions are very different in nature, yet are interrelated.

Two high quality still images of high spatial resolutions, but extremely low “ temporal resolution ” Video sequence has lower spatial resolution but a high temporal resolution. The goal is to construct a new sequence of high spatial and high temporal resolutions.

Increasing Space-Time Resolution in Video E. Shechtman, Y. Caspi, and M. Irani, European Conference on Computer Vision (ECCV), May Demos/SpaceTimeSR/SuperRes_demos.html

Michal Irani & shmuel Peleg

You know the relative displacements in image sequences You have some knowledge of the imaging process We can improve Image resolution Similar to back-projection used in tomography

Image resolution depends on the physical char ’ of the sensor Optics Density Spatial response Of the detector elements The iterative algo ’ to increase image resolution, Based on the problem of reconstruction 2-D object from its 1-D projections.

Do you remember Tomography (I hardly do)

In SR case, each low resolution pixel is a “ projection ” Of a region in the scene whose size is determined by the Imaging blur. (similar to back-projection)

Accurate knowledge of the relative scene locations sensed by each pixel in the observed images is necessary for SR. We assume that local motion can be described by translations and rotations only.

Obtaining the parameters of the IM Displacement Of the input images Blur

Image Registration Horizontal shift a, Vertical shift b, Rotation angel θ g2(x,y)=g1(xcosθ-ysinθ+a, ycosθ+xsinθ+b). Were obtain under the assumption which are valid for small displacements.

Iterative Refinement If the displacement between g1 and g2 isn ’ t small the we use the iterate process : 1.Initially assume no motion between the frames. 2.Compute motion parameters (by solving 2), add the computing motion to the existing motion estimate. 3.Warp frame g2 towards g1 using the current motion estimates, return to step 2 with the warped image g2. g2 gets closer to g1 at every iteration, The process terminates when the corrections to (a,b,θ) approach zero.

Recovering the Blur The image were blurred and we have to know something about the blur: Blur filter, frequency domain

(Iterative algorithm) Starting with an initial guess H for the high resolution image. The imaging process is simulated Set of low resolution images {g } (0) k If H were the correct high resolution image, {g } identical to images {g } (0) k k

Initial guess H for the high resolution image : Registering all the low resolution images over a fixed finer grid. Each high-resolution pixel in the finer grid is taken to be the average of all the low resolution pixels stacked above it.

H g 3 g 2 g 1 g k g 2 (0) g 1 g 3 g k BT 1 BT 2 BT 3 BT k Blur Transformation

Super resolution from three input real images: (a) (b) (C) (a) One of the 3 original images. (b) Initial guess. (c) Improved resolution image.

Super Resolution From Image Sequences Michal Irani & Shmuel Peleg