EE565 Advanced Image Processing Copyright Xin Li Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

EE565 Advanced Image Processing Copyright Xin Li Image Interpolation  Image Interpolation Importance of geometry Edge directed interpolation PDE-based techniques  Applications Super-resolution Inpainting (error concealment) Compressive sensing

EE565 Advanced Image Processing Copyright Xin Li Importance of Geometry  It is geometry that distinguishes image signals (2D) from audio and speech (1D)  2D image is the projection of 3D geometry (a seemingly-trivial statement)  Geometric elements and properties: edge (location and orientation), object (convexity) … …

EE565 Advanced Image Processing Copyright Xin Li Geometric Constraint of Edges 3D visualizationsingle-edge image Along the edge orientation, intensity field is homogeneous Across the edge orientation, intensity field evolves fast Observations:

EE565 Advanced Image Processing Copyright Xin Li Why geometry is difficult?  Geometry is embedded in the array of image pixels, which is not straightforward to be exploited by any linear operations (filtering or transform)  Image signals form a manifold in an extremely high-dimensional space (What is a manifold? Think of a string in the real world, it is a 3D object and a 1D manifold)

EE565 Advanced Image Processing Copyright Xin Li Interpolation Problem down sampling original high-resolution image X low-resolution image Y interpolated high-resolution image Z How to make Z as close to X as possible?

EE565 Advanced Image Processing Copyright Xin Li bilinear Interpolation Review (e+w)/2 (n+s)/2 (ne+nw+se+sw)/4

EE565 Advanced Image Processing Copyright Xin Li Why bilinear is bad?  Edge blurring  Jagged artifacts X Z Jagged artifacts X Z Edge blurring

EE565 Advanced Image Processing Copyright Xin Li Where does image quality degradation come from?  Violation of geometric constraint  Keep in mind an edge could be arbitrarily oriented Edge blurringJagged artifacts Along the edge orientation, intensity field is inhomogeneous Across the edge orientation, intensity field evolves slower

EE565 Advanced Image Processing Copyright Xin Li Ideas to Do Better  Recognize the importance of edges Location: where are they? Orientation: how are they oriented?  Identify appropriate mathematical tools for exploiting geometric constraints Heuristics Statistical vs. PDE

EE565 Advanced Image Processing Copyright Xin Li Heuristics: Edge Location  Intuitively, edges are characterized by fast intensity value variation (i.e., large gradient)  To avoid edge blurring, we need to spatially adaptively adjust interpolation coefficients Question: how to incorporate gradient cue into a spatially adaptive interpolation scheme?

EE565 Advanced Image Processing Copyright Xin Li Desired Improvement XYZ bil Z esi horizontal edge vertical edge

EE565 Advanced Image Processing Copyright Xin Li Here Comes the Trick Step 1: interpolate the missing pixels along the diagonal black or white? Step 2: interpolate the other half missing pixels a b cd Since |a-c|=|b-d| x x has equal probability of being black or white a b c d Since |a-c|>|b-d| x=(b+d)/2=black x

EE565 Advanced Image Processing Copyright Xin Li Edge Sensitive Interpolation (ESI) b c d x a If |a-c|>|b-d| x=(b+d)/2 If |a-c|<|b-d| x=(a+c)/2 If |a-c|=|b-d| x=min or max a b cd x In simple words, we want to avoid interpolate along the direction which has a large gradient

EE565 Advanced Image Processing Copyright Xin Li Experiment Result Bilinear interpolationEdge sensitive interpolation

EE565 Advanced Image Processing Copyright Xin Li Heuristics: Edge Orientation  Can we do better? Yes! Gradient is only a first-order characteristics of edge location ESI makes binary decision with two orthogonal directions  How to do better? We need some mathematical tool that can work with arbitrary edge orientation

EE565 Advanced Image Processing Copyright Xin Li Motivation x y Along the edge orientation, We observe repeated pattern (0,0) (-1,2) (-2,4) (1,-2) : :.. pattern

EE565 Advanced Image Processing Copyright Xin Li Geometric Duality same pattern down sampling

EE565 Advanced Image Processing Copyright Xin Li Bridge across the resolution High-resolution Low-resolution 2i 2j 2i+2 2i-2 2j-22j+2 Cov(X 2i,2j,X 2i+k,2j+l )≈Cov(X 2i,2j,X 2i+2k,2j+2l ) (k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}

EE565 Advanced Image Processing Copyright Xin Li Least-Square (LS) Method  Solve over-determined system  Solve square linear system

EE565 Advanced Image Processing Copyright Xin Li LS-based estimation X1X1 X2X2 X3X3 X4X4 X5X5 X6X6 X7X7 X8X8 X For all pixels in 7x7 window, we can write an equation like above, which renders an over-determined system with 49 equations and 8 unknown variables Use LS method to solve

EE565 Advanced Image Processing Copyright Xin Li Step 1: Interpolate diagonal pixels -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

EE565 Advanced Image Processing Copyright Xin Li Step 2: Interpolate the Other Half -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

EE565 Advanced Image Processing Copyright Xin Li Experiment Result bilinearEdge directed interpolation

EE565 Advanced Image Processing Copyright Xin Li After Thoughts  Pro Improve visual quality dramatically  Con Computationally expensive  Further optimization Translation invariant derivation of interpolation coefficients a’s