SuperResolution (SR): “Classical” SR (model-based) Linear interpolation (with post-processing) Edge-directed interpolation (simple idea) Example-based SR (data-driven) Early attempt (limited success) Reinvented with sparse coding* Latest advance: local self-example Toward a better understanding of self- similarity

2 Why bilinear is bad? Edge blurring Jagged artifacts X Z X Z Edge blurring

3 Heuristics: Edge Orientation Can we do better? Yes! Gradient is only a first-order characteristics of edge location ESI makes binary decision with two orthogonal directions How to do better? We need some mathematical tool that can work with arbitrary edge orientation

4 Motivation x y Along the edge orientation, We observe repeated pattern (0,0) (-1,2) (-2,4) (1,-2) : :.. pattern

5 Geometric Duality same pattern down sampling

6 Bridge Across the Resolution High-resolution Low-resolution 2i 2j 2i+2 2i-2 2j-22j+2 Cov(X 2i,2j,X 2i+k,2j+l )≈Cov(X 2i,2j,X 2i+2k,2j+2l ) (k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}

7 Step 1: Interpolate diagonal pixels -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

8 Step 2: Interpolate the Other Half -Formulate LS estimation problem with pixels at low resolution and solve {a 1,a 2,a 3,a 4 } -Use {a 1,a 2,a 3,a 4 } to interpolate the pixel 0 at the high resolution Implementation:

9 Experiment Result bilinearEdge directed interpolation

10 After Thoughts Pro Improve visual quality dramatically Con Computationally expensive (used to be) Not suitable for corners/textures Further optimization Translation invariant derivation of interpolation coefficients a’s

Example-Based Super Resolution Training Set

Nearest-Neighbor Search

Kd-trees* The kd-tree is a powerful data structure that is based on recursively subdividing a set of points with alternating axis-aligned hyperplanes. The classical kd-tree uses O(dn lgn) precomputation time, O(dn) space and answers queries in time logarithmic in n, but exponential in d l5l5 l1l1 l9l9 l6l6 l3l3 l 10 l7l7 l4l4 l8l8 l2l2 l1l1 l8l8 1 l2l2 l3l3 l4l4 l5l5 l7l7 l6l6 l9l

Example-based SR Algorithm

Experimental Results LR input

SR via Sparse Representation* How should we regularize the super-resolution problem? Markov random field [Freeman et. Al. IJCV ‘00] Primal sketch prior [Sun et. Al. CVPR ‘03] Neighbor embedding [Chang et. Al. CVPR ‘04] Soft edge prior [Dai et. Al. ICCV ‘07] ? Basic Idea: High-resolution patches have a sparse linear representation with respect to an overcomplete dictionary of patches randomly sampled from similar images (turning SR into a Compressed Sensing problem) output high-resolution patchhigh-resolution dictionary for some with

SR via Local Self-Example /lss_upscale/index.htmlhttp:// /lss_upscale/index.html(Siggraph’2011)

MRI Basics K-space IFT FT

What is Compressed Sensing? FT 22 radial lines in Fourier domain (perfect reconstruction is achieved) Candes, E.J.; Romberg, J.; Tao, T.;, "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information," Information Theory, IEEE Transactions on, vol.52, no.2, pp , Feb Basic idea: the magic of L 1

The Idea of Convex Relaxation convex relaxation convex nonconvex

CS via Alternating Projection Projection onto Observation Constraint set Projection onto Regularization Constraint set

What is Your Attack? “All roads lead to Rome.” Can PDE model be used for CS? Can wavelet model be used for CS? Can other patch-based model be used for CS? MRI image reconstruction For phantom 256x256 image, 8 radial lines are sufficient for PR

Experimental Results sparse_11_evolution.gif BM3D-CS

Summary of Part II Relationship to Part I Difference: likelihood P(Y|X) or data term Same: prior P(X) or regularization E(u) PDE vs. wavelet vs. patch Wise craftsman never blame tools Local view vs. nonlocal view Self-similarity in space and time