Week 11 – Spectral TV and Convex analysis Guy Gilboa Course

Topics: Some basic definitions in convex analysis. New research on Spectral TV. Guy Gilboa, Technion2

Classical Fourier filtering example Guy Gilboa, Technion3

Classical Signal Processing (Fourier) Positive features: Decomposition (transform) into a better representation, orthonormal basis. Filtering in the transform space – simple amplification or attenuation of coefficients. Spectrum plot – visualization of active frequencies, L2 energy is preserved – Perseval identity. Linearity – forward and inverse transforms are linear. A well established mathematical theory and fast computational methods. Known drawbacks : Does not handle well discontinuities and spatially local features, not an adequate basis for images.. Guy Gilboa, Technion4

Variational Spectral Processing Decomposition (transform) into a better representation, orthonormal basis. Filtering in the transform space – simple amplification or attenuation of coefficients functions. Spectrum plot – visualization of active “generalized- frequencies”, Perseval-type rule. Linearity – forward and inverse transform are is linear. A well established mathematical theory and fast computational methods. Not yet.. Guy Gilboa, Technion5

TV spectral representation [G., SIAM-IS, 2014] Let u(t) be the TV-flow solution at time t with u(0)=f. TV- flow f t S(t)f... ϕ t H(t) 1 0 S(t) Guy Gilboa, Technion6

Nonlinear eigenvalue problem Linear problem (L linear operator) General operator T: A convex functional J(u) induces an operator p(u) by its subgradient: Guy Gilboa, Technion7

Understanding a regularizer is knowing its eigenfunctions [Alter-Caselles-Chambolle-2003]. My view : What are the TV eigenfunctions? Guy Gilboa, Technion8

Why do we get a delta in time for eigenfunctions ? Guy Gilboa, Technion9

Ideal low-pass-filter (LPF) eigenvalue Guy Gilboa, Technion10

Standard possible filters (borrowing the names from classical signal processing) Guy Gilboa, Technion11

1D Decomposition Example Guy Gilboa, Technion12

Application Guy Gilboa, Technion13

TV Band-Pass and Band-Stop filters fS(t) TV Band-stopTV Band-pass

Old man

Old man – close up, original

Old man – 2 modes attenuated

Wavelet vs. Spectral-TV decomposition f Haar WaveletsSpectral TV Guy Gilboa, Technion18

Haar Wavelets vs. Spectral TV WaveletSpectral TV Guy Gilboa, Technion19

Texture analysis and processing in the spectral TV domain with Dikla Horesh Guy Gilboa, Technion20

Spatially varying texture Perspective Lighting Combination Goal– decompose textures which are gradually varying in scale, contrast or lighting. Scale change Perspective Lighting

Spatially varying contrast and scale Guy Gilboa, Technion22 Input f(x) T(x)

What happens for a natural image? Guy Gilboa, Technion23

Proposed result How can we use it to separate? Classical TV-G separation at some cutoff scale Guy Gilboa, Technion24 Time of maximal value of phi(t;x) for each pixel x Input

Algorithm Input image Spectral decomposition Max response time Surface Fitting Take Max Φ in each pixel Separation Bands Separated image layers Values in percentiles are taken for surface fitting

Separation surface Band width taken for separationSeparation bands Input image Example 1

Decomposition example (perspective) Maximal phi response Proposed Input Rolling-Guidance-Filter [*] [*] Zhang et al, ECCV-2014 Guy Gilboa, Technion27

Desired textureStructure Input Texture enhancement

Application: Texture enhancement enhanced Input Attenuated

Application: Texture enhancement (2) Desired textureStructure Input

Enhancement 2 enhanced Input Attenuated Guy Gilboa, Technion31

Michael Moeller’s texture transfer Guy Gilboa, Technion32