Structured Face Hallucination Chih-Yuan Yang Sifei Liu Ming-Hsuan Yang Electrical Engineering and Computer Science 1

Outline Motivation Related work Proposed method Experimental results Conclusions 2

Motivation Generate high-quality face images 3 Algorithm

Challenges 4 How to effectively model a face? –Landmark points How to preserve the consistency of details? –Transfer details of a whole component –Maintain consistency of edges in upsampling –Exploit statistics of edge sharpness

Face Hallucination [Liu07] PCA on intensities –Global constraint MRF on residues –High-frequency details Bilateral filtering as post-processing –Suppress ghost effects 5

Sparse Representation [Yang08] NMF on intensity –Global constraint Patch mapping through a pair of sparse dictionaries –Restore the high-frequency details 6

Position Patch [Ma10] No global constraint Only local constraint by patch position –Only use exemplar patches at the same position –Weighted averaging exemplar patches 7

Proposed Approach Three classes –facial components Transfer the HR details from the whole region of a component –edges Preserve edge structures and restore sharpness by statistical prior –smooth regions Transfer the HR details from small patches

Aligning Component Exemplars Exemplar images are labeled Each component is aligned individually 9 Generate low-resolution exemplar images Search for the most similar exemplar

Insights Consistency –Consistent details because the whole component is transferred –The pair of eyes is considered as one component, as well as the eyebrows Effectiveness –Landmark points enable the comparison for a whole component –Effective for various shapes, sizes, and positions 10

Preserve Edge Structures Direction-Preserving Upsampling 11 Bilinear interpolation preserves the directional similarity in HR Regularize the HR image

Restore Edge Sharpness 12 upsamplededge centermag. of grad.enlarged restored

Smooth Regions Approach –Find the most similar LR patch and transfer the HR gradients Advantage –Highly adaptive Achieved by –PatchMatch algorithm –Low computational load Restriction –Consistency –Accuracy 13 Patch only Component Exemplar Patch only Edge Model and Priors

Generate Output Images 14 Merge gradient maps Generate output images

Experimental Results 15

16

17

Conclusions Structured face hallucination –Effective whole component exemplars –Preserved edge structures and robust statistical sharpness priors Preliminary results –Effective and consistent high-frequency details –Robustness 18