Milanfar et al. EE Dept, UCSC 1 “Locally Adaptive Patch-based Image and Video Restoration” Session I: Today (Mon) 10:30 – 1:00 Session II: Wed Same Time, Same Room Thank you for participating in and contributing to our mini- symposium on

Milanfar et al. EE Dept, UCSC 2 Local Adaptivity + Patch-Based Approaches State of the Art Performance A Convergence of Ideas Extremely Popular

Milanfar et al. EE Dept, UCSC 3 Patchy the Pirate Patch-based methods have become so popular in fact ….

Milanfar et al. EE Dept, UCSC 4 Multi-dimensional Kernel Regression for Video Processing and Reconstruction *Joint work with Hiro Takeda (UCSC), Mattan Protter and Michael Elad (Technion), Peter van Beek (Sharp Labs of America) SIAM Imaging Science Meeting, July 7, 2008 Peyman Milanfar* EE Department University of California, Santa Cruz

Milanfar et al. EE Dept, UCSC 5 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 6 Summary Motivation: –Existing methods make strong assumptions about signal and noise models. –Develop “universal”, robust methods based on adaptive nonparametric statistics Goal: –Develop the adaptive Kernel Regression framework for a wide class of problems, including video processing; producing algorithms competitive with state of the art.

Milanfar et al. EE Dept, UCSC 7 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 8 Kernel Regression Framework The data model A sample The regression function Zero-mean, i.i.d noise (No other assump.) The number of samples The sampling position The specific form of may remain unspecified for now.

Milanfar et al. EE Dept, UCSC 9 The data model Local representation (N-terms Taylor expansion) Note –With a polynomial basis, we only need to estimate the first unknown, –Other localized representations are also possible, and may be advantageous. Local Approximation in KR Unknowns

Milanfar et al. EE Dept, UCSC 10 Optimization Problem We have a local representation with respect to each sample: Minimization N+1 terms This term give the estimated pixel value at x. The regression order The choice of the kernel function is open, e.g. Gaussian.

Milanfar et al. EE Dept, UCSC 11 Locally Linear Estimator The optimization yields a pointwise estimator: The bias and variance are related to the regression order and the smoothing parameter: –Large N  small bias and large variance –Large h  large bias and small variance The weighted linear combinations of the given data Equivalent kernel function Kernel function The smoothing parameter The regression order

Milanfar et al. EE Dept, UCSC 12 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 13 (2D) Data-Adaptive Kernels Classic kernelData-adapted kernel Take not only spatial distances, but also radiometric distances (pixel value differences) into account Data-adaptive kernel function Yields locally non-linear estimators

Milanfar et al. EE Dept, UCSC 14 Simplest Case: Bilateral Kernels =. =. =. Low noise case Spatial kernel Radiometric kernel

Milanfar et al. EE Dept, UCSC 15 =. =. =. Bilateral Kernel (High Noise) Spatial kernel Radiometric kernel High noise case Effectively useless

Milanfar et al. EE Dept, UCSC 16 Better: Steering Kernel Method Local dominant orientation estimate based on local gradient covariance H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp , February 2007.

Milanfar et al. EE Dept, UCSC 17 Steering Kernel Kernel adapted to locally dominant structure The steering matrices scale, elongate, and rotate the kernel footprints locally. Local dominant orientation estimation Steering matrix ElongateRotateScale

Milanfar et al. EE Dept, UCSC 18 Steering Kernel Function Steering matrix estimation –Naive estimate –(Compact) Singular value decomposition Steering matrix With regularization

Milanfar et al. EE Dept, UCSC 19 Steering Kernel (Low Noise) Kernel weights and footprints: Low noise case FootprintsWeights Steering kernel as a function of x i with x held fixed Steering kernel as a function of x with x i and H i held fixed

Milanfar et al. EE Dept, UCSC 20 Steering Kernel (High Noise) High noise case FootprintsWeights Steering kernel as a function of x i with x held fixed Steering kernel as a function of x with x i and H i held fixed Steering approach provides stable weights even in the presence of significant noise. Kernel weights and footprints:

Milanfar et al. EE Dept, UCSC 21 Some Related (0 th -order) Methods Non-Local Means (NLM) –A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.” Multiscale Modeling & Simulation, 4(2): , Optimal Spatial Adaptation (OSA) –C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans. on Image Processing, 15(10): , Oct SKRNLMOSA

Milanfar et al. EE Dept, UCSC 22 Adaptive Kernels for Interpolation When there are missing pixels: –We cannot have the radiometric distance. –Using a “pilot” estimate, fill the missing pixels: Classic kernel regression Cubic or bilinear interpolation ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Milanfar et al. EE Dept, UCSC 23 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Regression in 3-D Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 24 Kernel Regression in 3-D Setup is similar to 2-D, but….. Data samples come from various (nearby) frames Signal “structure” is now in 3-D We can perform –Denoising –Spatial Interpolation –Frame rate upconversion –Space-time super-resolution Spatial gradients Temporal gradients

Milanfar et al. EE Dept, UCSC 25 Kernel Regression in 3-D Cont. Two ways to proceed –Adaptive Implicit-Motion Steering Kernel (AIMS) Roughly warp the data to “neutralize” large motions Implicitly capture sub-pixel motions in 3-D Kernel –Motion-Aligned Steering Kernel (MASK) Estimate motion with subpixel accuracy Accurately warp the kernel (instead of the data)

Milanfar et al. EE Dept, UCSC 26 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Regression in 3-D Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 27 AIMS Kernel in 3-D Steering kernel visualization examples A plane structureSteering kernel weights Isosurface A tube structure

Milanfar et al. EE Dept, UCSC 28 AIMS Motion Compensation Large displacements make orientation estimation difficult. By neutralizing the large displacement, the steering kernel can effectively spread again. Small motionsLarge motions The local kernel effectively spread along the local motion trajectory. Shift down Shift up The local kernel after motion compensation. Important: The compensation does not require subpixel accurate motion estimation, nor does it require interpolation

Milanfar et al. EE Dept, UCSC 29 AIMS Contains Implicit Motion “Small” motion vector Optical flow equation Assuming the patch moves with approximate uniformity Homogeneous Optical Flow Vector (Eigenvalues of C) Space-time gradients of roughly compensated data

Milanfar et al. EE Dept, UCSC 30 AIMS Summary AIMS is a two-tiered approach. 1. Neutralize whole-pixel motions D SKR with implicit subpixel motion information Steering matrices estimated from the motion compensated data in 3-D.

Milanfar et al. EE Dept, UCSC 31 A Simple Example of AIMS Original EIA imageLow resolution frame (9 frames with large motions) Lanczos (single frame) AIMS without motion compensation AIMS with motion compensation Deblurred using BTV

Milanfar et al. EE Dept, UCSC 32 Foreman Example Lanczos (frame-by-frame upscaling) AIMS Factor of 2 upscaling Input video (QCIF: 144 x 176 x 28)

Milanfar et al. EE Dept, UCSC 33 Spatial Upscaling Example Input (200 x 200) Upscaled image by AIMS (multi-frame, 5 frames), 400x400

Milanfar et al. EE Dept, UCSC 34 Spatiotemporal Upscaling Input video (200 x 200 x 20) Single frame steering kernel regression (400 x 400 x 20) Spatiotemporal classic kernel regression (400 x 400 x 40) AIMS regression (400 x 400 x 40)

Milanfar et al. EE Dept, UCSC 35 Outline Background and Motivation Classic Kernel Regression Data-Adaptive Regression Regression in 3-D Adaptive Implicit-Motion Steering Kernel (AIMS) Motion-Aligned Steering Kernel (MASK) Conclusions

Milanfar et al. EE Dept, UCSC 36 Motion-Aligned Steering Kernel Motion is explicitly estimated to subpixel accuracy Kernel weights are aligned with the local motion vectors using warping/shearing The warped kernel acts directly on the data –Handles large and/or complex motions “2-D motion-steered” (spatial) kernel 1-D (temporal) kernel Accurate, explicit motion estimates

Milanfar et al. EE Dept, UCSC 37 Intuition Behind the MASK 2-D “motion-steered” (spatial) kernel1-D (temporal) kernel

Milanfar et al. EE Dept, UCSC 38 The Shapes of MASK Spreads along spatial orientations and local motion vectors. Local data Slices of MASK kernels

Milanfar et al. EE Dept, UCSC 39 Implementation of MASK Block diagram Spatial gradient estimation Spatial orientation estimation Dense motion estimation Temporal gradient estimation MASK Input video Output video Spatial Steering matrix Local motion Note: The choice of motion estimation algorithm is not restricted to gradient-based methods

Milanfar et al. EE Dept, UCSC 40 A Comparison of AIMS and MASK Spin Calendar video Input video (200 x 200 x 20) AIMS (400 x 400 x 40) MASK (400 x 400 x 40)

Milanfar et al. EE Dept, UCSC 41 A Comparison of AIMS and MASK Foreman video Input video (QCIF: 144 x 176 x 28) AIMS + BTV deblurring (CIF: 288 x 352 x 28) MASK + BTV deblurring (CIF: 288 x 352 x 28)

Milanfar et al. EE Dept, UCSC 42 Conclusions We extended the 2-D kernel regression framework to 3-D. –Illustrated 2 distinct approaches AIMS: Avoids subpixel motion estimation, needs comp. for large motions MASK: Needs subpixel motion estimation, deals directly with large motions –Which is better? Depends on the application. The overall 3-D SKR framework is simultaneously well-suited for spatial, temporal, and spatiotemporal –upscaling, denoising, blocking artifact removal, superresolution –not only in video but in general 3-D data sets. Future work –Integration of deblurring directly in the 3-D framework –Computational complexity