A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013

In a Nutshell Spatial Input Transform Analysis Transform Filtering Spatial Output Fourier inspiration: Fourier Scale Spectral TV Flow TV Scale

Relations to eigenvalue problems General linear: ( L linear operator) Functional based Linear Nonlinear

What can a transform-based approach give us? Scale analysis based on the spectrum. New types of filtering – otherwise hard to design: nonlinear LPF, BPF, HPF. Nonlinear spectral theory – relation to eigenfunctions and eigenvalues. Deeper understanding of the regularization, optimal design with respect to data, noise and artifacts.

Examples of spectral applications today: Eigenfunctions for 3D processing Taken from Zhang et al, Spectral mesh processing, Taken from L Cai, F Da, Nonrigid deformation recovery.., 2012.

Image Segmentatoin Eigenvectors of the graph Laplacian [Taken from I. Tziakos et al, Color image segmentation using Laplacian eigenmaps, 2009 ]

Some Related Studies Andreu, Caselles, Belletini, Novaga et al – TV flow theory. Steidl et al 2004 – Wavelet – TV relation Brox-Weickert 2006 – scale through TV-flow Luo-Aujol-Gousseau 2009 – local scale measures Benning-Burger 2012 – ground states (nonlinear spectral theory) Szlam-Bresson – Cheeger cuts. Meyer, Vese, Osher, Aujol, Chambolle, G. and many more – structure-texture decomposition. Chambolle-Pock 2011, Goldstein-Osher 2009 – numerics.

Scale Space – a Natural Way to Define Scale Well talk specifically about total-variation (TV-flow, Andreu et al ): Scale space as a gradient descent:

TV-Flow: A behavior of a disk in time [Andreu-Caselles et al–2001,2002, Bellettini-Caselles-Novaga-2002, Meyer-2001] Center of disk, first and second time derivatives: t … …

Spectral TV basic framework Phi(t) definition

Reconstruction

Spectral response Spectrum S(t) as a function of time t: t S(t) f

Spectrum example fS(t)

Dominant scales

Eigenvalue problem The nonlinear eigenvalue problem with respect to a functional J(u) is defined by: Well show a connection to the spectral components.

Solution of eigenfunctions

What are the TV eigenfunctions? [Giusti-1978], [Finn-1979],[Alter-Caselles- Chambolle-2003].

Filtering Let H(t) be a real-valued function of t. The filtered spectral response is The filtered spatial response is H(t)

Filtering, example 1: TV Band-Pass and Band-Stop filters Band-pass Band-stop fS(t)

Disk band-pass example S(t)

We have the basic framework Spatial Input Transform Analysis Transform Filtering Spatial Output

Numerics Many ways to solve. Variational approach was chosen: Currently use Chambolles projection algorithm (some spikes using Split-Bregman, under investigation). In time: 2 nd derivative - central difference 1 st derivative - forward differnce Discrete reconstruction algorithm proved for any regularizing scale-space (Th. 4).

TV-Flow as a LPF

Nonlocal TV Reminder: NL-TV (G.-Osher 2008): Gradient Functional

Spectral NL-TV? The framework can fit in principle many scale-spaces, like NL-TV flow. We can obtain a one-homogeneous regularizer. What is a generalized nonlocal disk? What are possible eigenfunctions? It is expected to be able to process better repetitive textures and structures.

Sparseness in the TV sense Sparse spectrum – the signal has only a few dominant scales. Or many small ones (here TV energy is large) Can be a large objects Natural images – are not very sparse in general

Noise Spectrum Various standard deviations: S(t)

Noise + signal Not additive. Spreads original image spectrum. Needs to be investigated. u f f-u Band-pass filtered

Spectral Beltrami Flow? Initial trials on Beltrami flow with parameterization such that it is closer to TV Original Beltrami Flow Spectral Beltrami Difference images: Keeps sharp contrast Breaks extremum principle Values along one line (Green channel)

Segmentation priors Swoboda-Schnorr 2013 – convex segmentation with histogram priors. We can have 2D spectrum with histograms Use it to improve segmentation S(t,h)

Texture processing Many texture bands We can filter and manipulate certain bands and reconstruct a new image. Generalization of structure-texture decomposition.

Processing approach Deconstruct the image into bands Identify salient textures Amplify / attenuate / spatial process the bands. Reconstruct image with processed bands

Color formulation Vectorial TV – all definitions can be generalized in a straightforward manner to vector-valued images. Bresson-Chan (2008) definition and projection algorithm is used for the numerics.

Orange example

Orange – close up Original Modes 2,3=0 Modes 2-5=x1.5

Selected phi(t) modes (1, 5, 15, 40) residual f

Old man

Old man – close up Original 2 modes attenuated 7 modes attenuated

Old Man - First 3 Modes Modes: 1 2 3

Take Home Messages Introduction of a new TV transform and TV spectrum. Alternative way to understand and visualize scales in the image. Highly selective scale separation, good for processing textures. Can be generalized to other functionals.

Thanks! Refs. Google Guy Gilboa publications Preliminary ideas are in SSVM 2013 paper. Most material is in CCIT Tech report 803. Up-to-date and organized - submitted journal version – contact me.