1. EE565 Advanced Image Processing Copyright Xin Li 20081 Motivating Applications HDTV Internet video Artistic reproduction Widescreen movie

2. EE565 Advanced Image Processing Copyright Xin Li 20082 Image Interpolation  Image Interpolation Importance of geometry Edge directed interpolation PDE-based techniques  Applications Super-resolution Inpainting (error concealment) Compressive sensing

3. EE565 Advanced Image Processing Copyright Xin Li 20083 Importance of Geometry  It is geometry that distinguishes image signals (2D) from audio and speech (1D)  2D image is the projection of 3D geometry (a seemingly-trivial statement)  Geometric elements and properties: edge (location and orientation), object (convexity) … …

4. EE565 Advanced Image Processing Copyright Xin Li 20084 Geometric Constraint of Edges 3D visualizationsingle-edge image Along the edge orientation, intensity field is homogeneous Across the edge orientation, intensity field evolves fast Observations:

5. EE565 Advanced Image Processing Copyright Xin Li 20085 Why geometry is difficult?  Geometry is embedded in the array of image pixels, which is not straightforward to be exploited by any linear operations (filtering or transform)  Image signals form a manifold in an extremely high-dimensional space (What is a manifold? Think of a string in the real world, it is a 3D object and a 1D manifold)

6. EE565 Advanced Image Processing Copyright Xin Li 20086 Interpolation Problem down sampling original high-resolution image X low-resolution image Y interpolated high-resolution image Z How to make Z as close to X as possible?

7. EE565 Advanced Image Processing Copyright Xin Li 20087 bilinear Interpolation Review (e+w)/2 (n+s)/2 (ne+nw+se+sw)/4

8. EE565 Advanced Image Processing Copyright Xin Li 20088 Why bilinear is bad?  Edge blurring  Jagged artifacts X Z Jagged artifacts X Z Edge blurring

9. EE565 Advanced Image Processing Copyright Xin Li 20089 Where does image quality degradation come from?  Violation of geometric constraint  Keep in mind an edge could be arbitrarily oriented Edge blurringJagged artifacts Along the edge orientation, intensity field is inhomogeneous Across the edge orientation, intensity field evolves slower

10. EE565 Advanced Image Processing Copyright Xin Li 200810 Ideas to Do Better  Recognize the importance of edges Location: where are they? Orientation: how are they oriented?  Identify appropriate mathematical tools for exploiting geometric constraints Heuristics Statistical vs. PDE

11. EE565 Advanced Image Processing Copyright Xin Li 200811 Heuristics: Edge Location  Intuitively, edges are characterized by fast intensity value variation (i.e., large gradient)  To avoid edge blurring, we need to spatially adaptively adjust interpolation coefficients Question: how to incorporate gradient cue into a spatially adaptive interpolation scheme?

12. EE565 Advanced Image Processing Copyright Xin Li 200812 Desired Improvement XYZ bil Z esi horizontal edge vertical edge

13. EE565 Advanced Image Processing Copyright Xin Li 200813 Here Comes the Trick Step 1: interpolate the missing pixels along the diagonal black or white? Step 2: interpolate the other half missing pixels a b cd Since |a-c|=|b-d| x x has equal probability of being black or white a b c d Since |a-c|>|b-d| x=(b+d)/2=black x

14. EE565 Advanced Image Processing Copyright Xin Li 200814 Edge Sensitive Interpolation (ESI) b c d x a If |a-c|>|b-d| x=(b+d)/2 If |a-c|<|b-d| x=(a+c)/2 If |a-c|=|b-d| x=min or max a b cd x In simple words, we want to avoid interpolate along the direction which has a large gradient

15. EE565 Advanced Image Processing Copyright Xin Li 200815 Experiment Result Bilinear interpolationEdge sensitive interpolation

16. EE565 Advanced Image Processing Copyright Xin Li 200816 Heuristics: Edge Orientation  Can we do better? Yes! Gradient is only a first-order characteristics of edge location ESI makes binary decision with two orthogonal directions  How to do better? We need some mathematical tool that can work with arbitrary edge orientation

17. EE565 Advanced Image Processing Copyright Xin Li 200817 Motivation x y Along the edge orientation, We observe repeated pattern (0,0) (-1,2) (-2,4) (1,-2) : :.. pattern

18. EE565 Advanced Image Processing Copyright Xin Li 200818 Geometric Duality same pattern down sampling

19. EE565 Advanced Image Processing Copyright Xin Li 200819 Bridge across the resolution High-resolution Low-resolution 2i 2j 2i+2 2i-2 2j-22j+2 Cov(X 2i,2j,X 2i+k,2j+l )≈Cov(X 2i,2j,X 2i+2k,2j+2l ) (k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}

20. EE565 Advanced Image Processing Copyright Xin Li 200820 Least-Square (LS) Method  Solve over-determined system  Solve square linear system

21. EE565 Advanced Image Processing Copyright Xin Li 200821 LS-based estimation X1X1 X2X2 X3X3 X4X4 X5X5 X6X6 X7X7 X8X8 X For all pixels in 7x7 window, we can write an equation like above, which renders an over-determined system with 49 equations and 8 unknown variables Use LS method to solve

22. EE565 Advanced Image Processing Copyright Xin Li 200822 Step 1: Interpolate diagonal pixels -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

23. EE565 Advanced Image Processing Copyright Xin Li 200823 Step 2: Interpolate the Other Half -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

24. EE565 Advanced Image Processing Copyright Xin Li 200824 Experiment Result bilinearEdge directed interpolation

25. EE565 Advanced Image Processing Copyright Xin Li 200825 After Thoughts  Pro Improve visual quality dramatically  Con Computationally expensive  Further optimization Translation invariant derivation of interpolation coefficients a’s