1. Application (I): Impulse Noise Removal Impulse noise

2. Application (II): Predictive Coding var=4653, var=26.78

3. EE591b Advanced Image Processing Copyright Xin Li 2006 3 Lossless Image Compression Review of MED used in JPEG-LS from EE465 GAP in CALIC scheme Least-Square based edge directed prediction Intra-coding scheme adopted by H.264/JVT standard

4. EE591b Advanced Image Processing Copyright Xin Li 2006 4 Linear Prediction entropy coding discrete source X binary bit stream probability estimation P(Y) Y Recall: Predictive Coding Prediction residue sequence Y usually contains less uncertainty (entropy) than the original sequence X

5. EE591b Advanced Image Processing Copyright Xin Li 2006 5 2D Predictive Coding raster scanning order X m,n causal half-plane

6. EE591b Advanced Image Processing Copyright Xin Li 2006 6 Median Edge Detection (MED) Prediction x w nnw Key: MED use the median operator to adaptively select one from three candidates (Predictors #1,#2,#4 in slide 44) as the predicted value.

7. EE591b Advanced Image Processing Copyright Xin Li 2006 7 Another Way of Implementation x w nnw If else if Q: which one is faster? You need to find it out using MATLAB yourself

8. EE591b Advanced Image Processing Copyright Xin Li 2006 8 Numerical Examples 100 50 100 50 50 V_edge H_edge n=50,w=100, nw=100 n+w-nw=50 n=100,w=50, nw=100 n+w-nw=50 Note how we can get zero prediction residues regardless of the edge direction

9. EE591b Advanced Image Processing Copyright Xin Li 2006 9 Image Example Fixed vertical predictor H=4.67bpp Adaptive (MED) predictor H=4.55bpp

10. EE591b Advanced Image Processing Copyright Xin Li 2006 10 JPEG-LS (the new standard for lossless image compression)*

11. EE591b Advanced Image Processing Copyright Xin Li 2006 11 Lossless Image Compression Review of MED used in JPEG-LS from EE465 GAP in CALIC scheme Least-Square based edge directed prediction Intra-coding scheme adopted by H.264/JVT standard

12. EE591b Advanced Image Processing Copyright Xin Li 2006 12 Gradient Adjusted Prediction

13. EE591b Advanced Image Processing Copyright Xin Li 2006 13 Image Example MED predictor H=4.55bpp GAP predictor H=4.39bpp

14. EE591b Advanced Image Processing Copyright Xin Li 2006 14 Context-based, Adaptive, Lossless Image Codec (CALIC)

15. EE591b Advanced Image Processing Copyright Xin Li 2006 15 Context Quantization Context formulation Without quantization, we will have 256 8 different contexts (so-called “context dilution” problem) After binary context quantization, we will reduce the number of contexts to 2 8 =256

16. EE591b Advanced Image Processing Copyright Xin Li 2006 16 Lossless Image Compression Review of MED used in JPEG-LS from EE465 GAP in CALIC scheme Least-Square based edge directed prediction (EDP) Intra-coding scheme adopted by H.264/JVT standard

17. EE591b Advanced Image Processing Copyright Xin Li 2006 17 Motivation Behind EDP GAP appears ad-hoc due the following reasons  Only two directions are considered  Difficult to justify the thresholds (32 and 80) used to classify weak and strong edges  Fundamental limitation with local gradients Two potential improvements  Truly direction adaptive  From local to nonlocal prediction

18. EE591b Advanced Image Processing Copyright Xin Li 2006 18 Key Observation No matter which classification strategy we adopt (e.g., strong vs. weak, horizontal vs. vertical), the prediction result can be viewed as a linear weighted average of the local neighborhood Is there a systematic and provably optimal way of tuning the weighting coefficients?  Again, the idea of localization will fly again

19. EE591b Advanced Image Processing Copyright Xin Li 2006 19 Recall: Autoregressive (AR) Model M equations, N unknown variables

20. EE591b Advanced Image Processing Copyright Xin Li 2006 20 Least-Square Estimation M equations, N unknown variables

21. EE591b Advanced Image Processing Copyright Xin Li 2006 21 LS-based Training of AR Model within a Local Causal Neighborhood T T+1 T double rectangular window containing causal neighbors the nearest N causal neighbors

22. EE591b Advanced Image Processing Copyright Xin Li 2006 22 Edge Directed Property : edge pixels : non-edge pixels The LS method provides a convenient way of finding solution for edge pixels without explicitly picking them out The weights derived by edge pixels work for X i since it lives along the same edge no preferred direction unique preferred direction edge pixels dominates the Least-Square process direction adaptive prediction

23. EE591b Advanced Image Processing Copyright Xin Li 2006 23 Predictor Performance Comparison MAP(4.56bpp)GAP(4.40bpp)10th-order EDP(4.22bpp) Comparison of prediction residue images by MAP, GAP and EDP

24. EE591b Advanced Image Processing Copyright Xin Li 2006 24 Efficient Implementation(I) Inclusion-and-Exclusion: memory  complexity + + + + + + ------------ + + + -- (1) (2) straightforward implementation arithmetic operations (1) (2) 5N 2 =500 2(T+1)N 2 =1600

25. EE591b Advanced Image Processing Copyright Xin Li 2006 25 Efficient Implementation(II) Switching strategy: per pixel  per edge (speed-up ratio=262,144/25,870  10) activate LS to update a j when |e i |>T - LS-based adaptation only enhances prediction performance around edges - predictor optimized for an edge pixel also works for its neighboring pixels along the same edge

26. EE591b Advanced Image Processing Copyright Xin Li 2006 26 Performance of EDP CALICTMWscheme I airplane baboon lenna peppers barb barb2 boats goldhill average 3.743.603.75 4.11 4.42 4.32 4.53 3.83 4.39 4.41 4.01 4.33 4.08 4.46 3.75 4.38 4.32 3.91 4.25 4.08 4.38 3.61 4.27 4.23 11.43.8210.7 11.5 17.2 17.0 16.9 17.1 - 10.8 12.7 18.2 26.3 15.2 22.5 - 4.02 4.35 4.11 4.52 3.80 4.39 4. 35 time (seconds) scheme II EDP(N=6,T=6) time (seconds) 5.885.735.8111.65.8134.4 (seconds)(hours) Performance (bpp) comparison among CALIC, TMW and EDP

27. EE591b Advanced Image Processing Copyright Xin Li 2006 27 Lossless Image Compression Review of MED used in JPEG-LS from EE465 GAP in CALIC scheme Least-Square based edge directed prediction Intra-coding scheme adopted by H.264/JVT standard

28. EE591b Advanced Image Processing Copyright Xin Li 2006 28 A Glimpse into H.264 It is a video (not image) coding standard However, there is so-called I (Intra-coded) frame in video coding which does not involve any temporal prediction Therefore, I-frame coding is conceptually identical to image coding H.264 Intra-coding is a lossy scheme (though lossless extension is conceivable)

29. EE591b Advanced Image Processing Copyright Xin Li 2006 29 Intra-prediction Modes in H.264 The idea of directional prediction is obvious, but moreover, the prediction goes from local to nonlocal (a pixel can be used to predict four pixels along the specified direction)

30. EE591b Advanced Image Processing Copyright Xin Li 2006 30 Patch-based Prediction (Open Research Problem) p non-parametric sampling Input image Shouldn’t the prediction of P be based on nonlocal patches instead of local neighborhood?

31. EE591b Advanced Image Processing Copyright Xin Li 2006 31 Preliminary Result Patch-based, H=3.67bppEDP, H=4.43bpp It is going to be the next breakthrough in lossless image compression

32. EE591b Advanced Image Processing Copyright Xin Li 2006 32 Lossless Image Coding Summary A well-define objective: use as few bits as possible (MSE=0) From ad-hoc prediction to more systematic way of designing predictors which can exploit the fundamental dependency of image source It still has a long way to go

33. Application (III): Super-resolution SR

34. EE565 Advanced Image Processing Copyright Xin Li 2008 34 Heuristics: Edge Orientation Can we do better?  Yes!  Gradient is only a first-order characteristics of edge location  ESI makes binary decision with two orthogonal directions How to do better?  We need some mathematical tool that can work with arbitrary edge orientation

35. EE565 Advanced Image Processing Copyright Xin Li 2008 35 Motivation x y Along the edge orientation, We observe repeated pattern (0,0) (-1,2) (-2,4) (1,-2) : :.. pattern

36. EE565 Advanced Image Processing Copyright Xin Li 2008 36 Geometric Duality same pattern down sampling

37. EE565 Advanced Image Processing Copyright Xin Li 2008 37 Bridge across the resolution High-resolution Low-resolution 2i 2j 2i+2 2i-2 2j-22j+2 Cov(X 2i,2j,X 2i+k,2j+l )≈Cov(X 2i,2j,X 2i+2k,2j+2l ) (k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}

38. EE565 Advanced Image Processing Copyright Xin Li 2008 38 Least-Square (LS) Method Solve over-determined system  Solve square linear system

39. EE565 Advanced Image Processing Copyright Xin Li 2008 39 LS-based estimation X1X1 X2X2 X3X3 X4X4 X5X5 X6X6 X7X7 X8X8 X For all pixels in 7x7 window, we can write an equation like above, which renders an over-determined system with 49 equations and 8 unknown variables Use LS method to solve

40. EE565 Advanced Image Processing Copyright Xin Li 2008 40 Step 1: Interpolate diagonal pixels -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

41. EE565 Advanced Image Processing Copyright Xin Li 2008 41 Step 2: Interpolate the Other Half -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

42. EE565 Advanced Image Processing Copyright Xin Li 2008 42 Experiment Result bilinearEdge directed interpolation

43. EE565 Advanced Image Processing Copyright Xin Li 2008 43 After Thoughts Pro  Improve visual quality dramatically Con  Computationally expensive Further optimization  Translation invariant derivation of interpolation coefficients a’s

44. Application (IV): Image Denoising Noisy denoised Ref.: Hirakawa, K.; Parks, T.W., "Image denoising using total least squares," Image Processing, IEEE Transactions on, vol.15, no.9, pp.2730-2742, Sept. 2006

45. Unsettled Questions Why AR is more effective on speech than image? How to choose the order of AR model and the size of training window? How to handle the interference of outliers? When does AR fail (as a tool of dimensionality reduction)?