Super-Resolution for Images and Video Ryan Prendergast and Prof. Truong Nguyen Video Processing Group University of California at San Diego

Outline Overview and Applications Image Super-Resolution Video Super-Resolution Conclusions

Problem Overview Single image interpolation – Variety of methods Extend results to multiple still-images and video

Application Domains Astronomy Satellite imagery HDTV Video on mobile Low bit-rate video Streaming internet Classic cost/quality trade-off. Increased need for real-time solutions.

Outline Overview and Applications Image Super-Resolution – Degradation Model – Registration Problem – Techniques for Reconstruction Video Super-Resolution Conclusions

Degradation Model Each low-resolution (LR) image/frame is modelled as a degraded version of the desired high-resolution (HR) image/frame. Most models will be the same for each image: distortion, sampling, noise. Motion will usually be different for each image.

Distortion (blurring) Different sources: – Lens blur – Motion blur – Atmospheric blur Model with a linear point spread function (PSF) PSF estimated or assumed known.

Noise Additive White – Coding noise can have colored model Described with statistical model (spectral density)

Sampling (Resolution Reduction) Decrease sampling rate from HR to LR Frequency-domain: aliasing Large features maintained, details lost

Motion (still image) Changes in the position of the camera lead to changes in the sample locations. Different motion for each image provides new information, making high-quality super- resolution possible. Estimated through registration process. Motion for video more complicated (see later).

Registration Simple global model: affine transform y = [y 1 y 2 ] T x = [x 1 x 2 ] T (vectors for horizontal and vertical pixel positions) Translational motion: b Rotation and scaling: A

Simple Motion Estimation: Phase Correlation Estimates the translational motion Pure translational shift is linear phase in Fourier domain. Normalize magnitude to find phase-shift term. Inverse Fourier transform:  [n 1,n 2 ] will have peak corresponding to translational shift. Method robust through degradation.

Additional Motion Estimation Global rotation and scaling more complicated: – Phase correlation on log-polar mapping of images. – Other Fourier domain examination (rotation consistency) – Spatial domain examination (signal projections) More complicated motion needs more complicated models Temporal motion in video will require local motion models.

Registration Results Registration relates all LR images to each other. Pixel locations on a single 2D grid are determined. Each pixel is still corrupted by distortion and noise. Reconstruction seeks to determine 2D samples at HR gird locations (HR pixels).

Super-Resolution: Brief Review of Methods Direct spatial domain processing. – Good: simple, direct, and efficient. – Bad: often low-quality results, not robust to errors. Frequency domain processing. – Good: efficient detailed model, recent methods give high-quality – Bad: reconstruction requires inflexible global models Bayesian solutions. – Good: adjust statistical model for content, can achieve high quality – Bad: computation, need accurate model details for best results Learning-based methods. – Good: High accuracy with good learning data – Bad: Model dependent on high complexity learning. Iterative techniques. – Good: improved reconstruction, careful iterations for more robust result – Bad: very slow, improper iteration parameters can make result diverge. Note: these methods not necessarily exclusive!

Direct Spatial-Domain Super- Resolution Can apply standard image interpolation techniques to non-uniformly sampled data (bilinear, bicubic). Matlab implementation function: griddata() Does not adjust result for noise and distortion. Separate deblurring phase necessary.

Spatial domain example Four LR images 3x3 decimation Nonuniform bicubic interpolation

Example (continued) Interpolated result does has distortion from PSF Deconvolution leads to ringing, same problem as in case of single image interpolation Interpolated ResultWith DeconvolutionOriginal

Iterative Regularization-Based Super-Resolution Regularization: examines ill-posed problem: no unique solution for unknown x. Well-posed solution found by introducing constraint: Constraint  for images: amplify unwanted features, like high-frequency noise. Minimizing solution will have reduced unwanted features.

Regularization (continued) Solution proposed by [1]: x is HR image, y k are LR observations, A k are known degradations (blur, sampling). Regularization  BTV is based on L 1 norm, shown to preserve edge features. L 1 similarity measure used. Requires iterative solution implementation using steepest decent. [1] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multi-frame super- resolution,” IEEE Trans. Image Processing, vol. 13, pp , Oct

Regularization Example Over-regularization Under-regularization Four images 3x3 decimation L 2 regularization Good result

Example (continued) L1 regularization Regularization tends to work better with oversampling scenario (more LR images) Over-regularization Under-regularization

Filter Bank Model for Super-Resolution Analysis bank: degradation model, translational motion only. Synthesis bank: determines linear minimum mean-squared error (LMMSE) solution for analysis bank. Requires spectral density model for HR image. Non-iterative Fourier domain implementation.

Required Model Information Distortion models Additive noise statistical models Global translational motion model HR image power spectral density (PSD) model: – Assumes stationary signal – Estimated from LR images

Image PSD Characteristics Peak at origin. Steep radial decay, varies with angular direction. Examine logarithmic scaling for decay details.

PSD Model Selection True HR PSD unknown, estimate from LR. Use parametric model to find estimate. A common model: Isotropic (rotationally invariant). Improve model with rotationally variant version:

Examine the measured spectrum in polar coordinates. Determine the radial and angular average functions. Numerical integration Measurements don’t contain details, but do indicate trends. Radial trend found in low-frequency component.

Zone Measurements Divide spectrum into several small zones. Less variance over each zones. Determine single measurement of  for each zone. Interpolate complete  function from sampled measurements. Numerical parameter selection of each  Parametric spectral estimation robust against distortion, noise, and even aliasing.

Reconstruction With all models for image and degradation known, linear reconstruction solution is found. Lab assignment provides implementation for LMMSE filter bank reconstruction.

Reconstruction Example Synthetic SR scenario: – Source image is corrupted and down-sampled by factor of (4,4). – Four source LR images used to reconstruct original (75% sample loss). – Blurring and noise introduced. Assumed unknown. – Precise registration known. Compare several reconstructions: – Single source image bicubic interpolation. – Proposed MMSE reconstruction using true HR PSD. – Proposed MMSE reconstruction using model PSD from interpolated set. – Direct spatial cubic interpolation based on Delaunay triangulation. – UCSC approach (Milanfar, et al.). Iterative joint interpolation and deblurring for L 2 norm minimization with Tikhonov regularization term to penalize high-frequency content (under-determined scenario, regularization necessary). 50 iterations.

OriginalBicubic interpolation (single frame)MMSE, true PSD MMSE, model PSDDelaunay TriangulationIterative, L2 regularization

PSNR Performance Vary noise and Gaussian distortion. Assume noise and distortion levels unknown. MMSE solution works best with known PSD MMSE with Estimated PSD similar performance to alternatives.

Second Example Two LR images, acquired through consumer digital camera. Global translational motion model. Resolution increase 2x2. No HR image to compare with. Single LR image

Single Frame Bicubic Interpolation

L 1 Regularization 2 Frames

LMMSE Interpolation 2 Frames

Outline Overview and Applications Image Super-Resolution Video Super-Resolution – Differences from still-image scenario – Degradation model – MMSE reconstruction Conclusions

Image  Video Super-Resolution New considerations: – Stationary/shift-invariant models don’t work well. Need local modelling for degradation. – Need to consider temporal consistency: quality of sequence as a whole, rather than only individual frame quality. – Temporal motion exists: reconstruction method must allow for arbitrary motion modelling and occlusion. – An increased in registration error should be expected at the reconstruction process.

Motion: Still-Image vs. Video Still-image scenario – Static scene. – Differences in observations results from camera motion. – Scene represented by 2D function of intensity: x(t 1,t 2 ). Video scenario – Dynamic scene. – Intensity function x(t 1,t 2 ) changes with time. – Complete motion model should account for both: Positional changes of acquisition device. Temporal motion of scene content.

Video Registration Difficulties Registration is most complicated part of modelling phase. Small errors in registration can cause significant reconstruction artifacts. Some level of registration error must be considered unavoidable. Example block-based motion model with registration error:

Effects of bad registration Super-resolution from 2 frames of video sequence. Block-based motion model leads to registration error. With no registration error compensationUsing concealment for poorly-registered data

Each frame has unique degradation (typically only motion differences). New motion model required for each LR frame.

Registration tends to “ignore” the PSF, and gives pixel locations based on motion and sampling. Considering multiple frames, it is more convenient to replace the true degradation model: with: Frame PSF will usually change to commute with motion. We consider all data on a single nonuniformly sampled 2D function, but the PSF for each pixel will vary.

Simplest when motion and PSF commute, equivalent models with PSF 1 = PSF 2. Purely translational motion, shift-invariant PSF. Rotational motion, rotation-invariant PSF. Further simplification if common PSF shared for all frames (reasonable assumption in many video scenarios).

Equivalent to a simple degradation system consisting of simply distortion, nonuniform sampling, and noise. Simplicity of this equivalent model ideal for scenarios where motion and PSF commute (Hardie 2007). However, real video is not accurately modelled with uch simple motion.

Consider desired HR pixels y and LR data points x (vectors of pixels). Wiener solution applied to small block of data for reconstruction. Distortion and noise models are contained inside the statistics of R xx and R xy. Local statistical and degradation models. Similar to non-uniform edge directed interpolation. MMSE Solution Approach

Example: 2x2 Video Super-Resolution Block based motion estimate (many registration errors). Consider eleven frames: 5 each preceding and following the current frame. Pixels with bad registration are detected and modelled as more noisy. Compare results with MATLAB implemented Delaunay-based cubic interpolation. n n+5 n-5

Proposed, 1 frame, dBProposed, 11 frames, dB Cubic interp., 1 frame, dBCubic interp., 11 frames, dB

Test conclusions: – Proposed method found PSNR and quality improvement with additional frames. – Effects of bad registration were reduced in MMSE approach. – Cubic interpolation performed at similar level for single frame, but significantly worse with multiple frames (not robust). Current SR bottleneck: inaccurate local registration.

Ongoing work in Super-Resolution Improving the motion model, reducing registration errors. Better modelling of content and degradation. Improving computational efficiency. Improving reconstruction quality. For video: considering multiple-frame reconstruction simultaneously for better temporal consistancy.

Preview: Lab Assignment Assignment will work with LMMSE image super-resolution: 1.Single-image LMMSE interpolation, spectral analysis 2.Image parametric spectral modelling 3.Multi-image LMMSE super-resolution, ideal case 4.Phase correlation registration, super-resolution in non-ideal cases 5.Final super-resolution, 7 noisy LR images. Spectrum and registration unknown.