1. Soft Edge Smoothness Prior for Alpha Channel Super Resolution Shengyang Dai 1, Mei Han 2, Wei Xu 2, Ying Wu 1, Yihong Gong 2 1.EECS Department, Northwestern University, Evanston, IL 2.NEC Laboratories America, Inc., Cupertino, CA

2. Prior is needed Minimize the reconstruction error  Degraded HR → LR input  Efficient solution by Back-projection (Irani’93) However …  SR is under-determined (Baker&Kanade’02, Lin&Shum’04)  Image prior is needed for regularization

3. What kind of prior? LR input Back-projection Bicubic interpolation Smooth edges are preferred

4. What kind of smoothness? binary value real value Hard edge vs. Soft edge Soft smoothness is preferred LR inputHard + Smooth Soft + Smooth

5. Questions  How to describe an edge?  How to obtain a soft smooth edge? Our solutions  Alpha channel edge description  Soft edge smoothness prior

6. Questions  How to describe an edge?  How to obtain a soft smooth edge? Our solutions  Alpha channel edge description  Soft edge smoothness prior

7. Image synthesis Alpha matting - 1 + alpha matting A closed form solution with smoothness assumption (Levin etc. ’06)

8. LR patches HR patches Matting for SR alpha matting convolution down-sampling synthesis

9. LR patches HR patches alpha matting convolution down-sampling synthesis SR? alpha matting SR synthesis Matting for SR Bicubic

10. Matting for SR ? SR

11. Questions  How to describe an edge?  How to obtain a soft smooth edge? Our solutions  Alpha channel edge description  Soft edge smoothness prior

12. Objective Regularity term prefers soft smooth edge Hard edgeSoft edge GeocutOur method

13. Objective Hard edgeSoft edge GeocutOur method

14. Pixel neighborhood

15. Image grid graph

16. Hard edge smoothness Geocut (Boykov&Kolmogorov’03) Curve C Cut metric: 1 0 : Neighborhood order of the image grid : Edge weight of the grid graph : Set of the pixels pairs of order k : Binary indication function on grid

17. Hard edge smoothness A regularity term prefers hard smooth edge Application  Segmentation problem  Reducing the metrication artifact Euclidean lengthCut metric

18. Objective Hard edgeSoft edge GeocutOur method

19. Soft edge smoothness Soft edge Level lines Soft edge is equivalent to a set of image level lines

20. Soft edge smoothness GeocutOur method Binary indication functionReal-valued function Cut metric Soft cut metric

21. Soft edge smoothness A regularity term prefers soft smooth edge Application  Super resolution  Reducing the jaggy effect

22. For SR Objective function Efficient optimization by steepest descent Critical parameters Two options  Alpha channel SR for each edge segment  Color channels SR over entire image …

23. Effect of …… only upper plane is shown Zoom factor Image size

24. Effect of Back-projection Note: color channels are processed separately

25. LR input Our result Bicubic interpolation Color channels SR

26. Color channels vs. alpha channel Color channels SR on the entire image Alpha channel SR on edge segments

27. Corner detection (He etc. 04) Alpha channel SR Process each edge segment separately

28. Comparison Bicubic Our method Back-projection

29. Comparison – reconstruction based Bicubic Our method Back-projection

30. Comparison – exemplar based Our method Learning low level vision Neighbor embedding (courtesy to Bill Freeman) (courtesy to Dit-Yan Yeung)

31. Conclusion Soft smoothness prior  With specific geometric explanation  Applicable to super resolution Alpha channel super resolution Limitation  The smoothness prior may not hold for texture