Kartic Subr Cyril Soler Frédo Durand Edge-preserving Multiscale Image Decomposition based on Local Extrema INRIA, Grenoble Universities MIT CSAIL

Multiscale image decomposition + + Medium Pixels Intensity Input Fine Coarse 1D

Motivation Detail enhancement Separating fine texture from coarse shading

What is detail?

Some examples

Related work Linear multiscale methods Edge-preserving approaches 1D Signal analysis

Related work : Linear multiscale methods Edge-preserving approaches 1D Signal analysis [Burt and Adelson 93] [Rahman and Woodell 97] [Pattanaik et al 98] [Lindeberg 94] Edges not preserved (Causes halos while editing)

Related work : Edge-preserving methods 1D Signal analysis [Farbman et al 08] [Fattal et al 07] [Bae et al 07] [Chen et al 07] Edge-aware Assume detail is low contrast

Related work: Empirical mode decomposition Linear multiscaleEdge-preserving approaches [Huang et al 98] Developed for 1D signals Detail depends on spatial scale Not edge-aware

Related work Edge-preserving approaches 1D Signal analysis Linear multiscale methods

Existing edge-preserving image decompositions Input Edge-preserving smoothing (e.g. bilateral filter) Base layer Detail layer (Input – Base) + Iteratively smooth input Recursively smooth base layer OR

Input Base layer Detail layer (Input – Base) + Edge-preserving smoothing (e.g. bilateral filter) Edge (preserved) Detail (smoothed) Existing edge-preserving image decompositions Assume detail is low-intensity variation

Challenge: Smoothing high-contrast detail Input

Challenge: Smoothing high-contrast detail Edge Low-contrast detail High-contrast detail

Conservative smoothing (bilateral filter with narrow range-Gaussian) Challenge: Smoothing high-contrast detail Edge preserved? Low-contrast detail smoothed? High-contrast detail smoothed?

Challenge: Smoothing high-contrast detail Edge preserved? Low-contrast detail smoothed? High-contrast detail smoothed? Aggressive smoothing (bilateral filter with wide range-Gaussian)

Example: Smoothing high-contrast detail Input[Farbman et al 2008] λ= 13, α = 0.2 [Farbman et al 2008] λ= 13, α = 1.2 Detail not smoothed Coarse features smoothed Edge smoothed

Our approach: Use local extrema Input Local maxima Local minima Detail = oscillations between local extrema

Our approach: Use local extrema Base = Local mean of neighboring extrema

Our approach: Use local extrema Local mean of neighboring extrema Edge preserved? Low-contrast detail smoothed? High-contrast detail smoothed?

Our detail extraction Input Base layer Detail layer + High-contrast detail smoothed Edges preserved

Algorithm Identify local extrema Estimate smoothed mean Detail at multiple scales Input: Image + number of layers

Algorithm: Illustrative example

Algorithm: Identifying local extrema Extrema detection kernel Local maxima Local minima

Algorithm: Estimating smoothed mean 1) Construct envelopes Minimal envelope Interpolation preserves edge [Levin et al 04] Maximal envelope

Algorithm: Estimating smoothed mean 2) Average envelopes Estimated mean

Algorithm: After one iteration + Input Base Detail

Algorithm: Mean at coarser scale Local maxima Local minima Widen extrema detection kernel

Algorithm: Mean at coarser scale Minimal envelope Maximal envelope

Algorithm: Mean at coarser scale Estimated mean

Recap: Detail extraction Identify local extrema Construct envelopes Average envelopes - Input Detail Smoothed mean (Base)

Identify local extrema Construct envelopes Average envelopes Recap: Detail extraction Smoothed mean Detail = Input - BaseBase Input

Base B 2 Base B 1 Input Detail D 2 Detail D 1 Recap: Multiscale decomposition Layer 1Layer 2Layer 3 Iteration 1 on input Iteration 2 on B 1 Recurse n-1 times for n-layers CoarseFine

Results

Results: Smoothing Input Smoothed

Results: Multiscale decomposition Medium Input Fine Coarse Low contrast edgeHigh contrast detailLow contrast edgeHigh contrast detail

Results: Multiscale decomposition Input

Results: Multiscale decomposition Fine Coarse

Results: Multiscale decomposition Input

Results: Multiscale decomposition After one iteration Base layerDetail layer

Results: Multiscale decomposition After two iterations Base layerDetail layer

Applications: Image equalization

Applications: Smoothing hatched images

Applications: Coarse illumination transfer

Applications: Tone-mapping HDR images

Comparison [Farbman et al 2008] Our Result

Our smoothing

Limitation Input Our Result

Conclusion Detail based on local extrema Smoothing high contrast detail Edge-preserving multiscale decomposition

Acknowledgements INRIA post-doctoral fellowship Equipe Associée with MIT ‘Flexible Rendering’ Adrien Bousseau & Alexandrina Orzan HFIBMR grant (ANR-07-BLAN-0331) Anonymous reviewers

C++ source: