Sampling and Interpolation: Examples from the real world Martin Vetterli with Benjamin Bejar Haro, Mihailo Kolundzija and R.Parhizkar

1. Sparse Sampling Classic versus new results: Known versus unknown locations Finite rate of innovation and performance bounds The price of discretization 2. Sampling physics: Wave equation Sampling physical fields resulting from PDEs The plenacoustic function and sampling wave fields Wavefield synthesis Sampling the plenoptic field 3. Sampling fields: Trajectory sampling Spatial fields and mobile sampling Sampling along trajectories 5. Conclusions Outline 2

3 Classic Sampling Case [WNKWRGSS, ] If a function x(t) contains no frequencies higher than W cps, it is completely determined by giving its ordinates at a series of points spaced 1/(2W) seconds apart. [if approx. T long, W wide, 2TW numbers specify the function] It is a representation theorem: – sinc(t-n) is an orthobasis for BL[- ,  ] – x(t)  BL[- ,  ] can be written as – Linear subspace, Shift-invariant Notes: – Slow convergence of the sinc series – Shannon-Bandwidth and information rate – Bandlimited is sufficient, not necessary! (e.g. bandpass sampling) – Many variations (irregular, multichannel, SISS, etc)

4 Classic Case: Subspaces Non uniform Kadec 1/4 theorem Derivative sampling Sample signal and derivative… … at half the rate Stochastic Power spectrum is good enough Shift-invariant subspaces Generalize from sinc to other shift-invariant Riesz bases (ortho. and biorthogonal) Structurally, it is the same thing! Multichannel sampling Known shift: easy Unknown shift: interesting (super-resolution imaging)

5 Classic Case: Subspaces Shannon bandlimited case or 1/T degrees of freedom per unit time But: a single discontinuity, and no more sampling theorem… Are there other signals with finite number of degrees of freedom per unit of time that allow exact sampling results?

6 Classic Case: Subspaces Shannon bandlimited case or 1/T degrees of freedom per unit time But: a single discontinuity, and no more sampling theorem… Are there other signals with finite number of degrees of freedom per unit of time that allow exact sampling results?

7 Examples of Non-bandlimited Signals

8 Classic Case and Beyond… Sampling theory beyond Shannon? – Shannon: bandlimitedness is sufficient but not necessary – Shannon bandwidth and shift-invariant subspaces: dimension of subspace Is there a sampling theory beyond subspaces? – Finite rate of innovation: Similar to Shannon rate of information – Non-linear set up – Position information is key! Thus, develop a sampling theory for classes of non-bandlimited but sparse signals! t x(t) t0t0 x1x1 t1t1 x0x0 … x N-1 t N-1 Generic, continuous-time sparse signal

9 Signals with Finite Rate of Innovation The set up: For a sparse input, like a weighted sum of Diracs – One-to-one map y n  x(t)? – Efficient algorithm? – Stable reconstruction? – Robustness to noise? – Optimality of recovery? sampling kernel

10 A simple exercise in Fourier series Periodic set of K Dirac pulses – Is not bandlimited! – Has a Fourier series Fourier integral leads to – K complex exponentials – Exponents depends on location – Weight depends on If we can identify the exponents – Diracs can be recovered – Weights are a linear problem (given the locations!) Note: classic result, back to Prony (1794), spectral estimation etc

11 A Representation Theorem Theorem [VMB:02] Given x(t), a periodic set of K Diracs, of period , weights {x k } and locations {t k }. Take a Dirichlet sampling kernel of bandwidth B, with B  an odd integer > 2K Then the N samples, N > B  T =  are a sufficient characterization of x(t). The key to the solution: Separability of non-linear from linear problem

The signal is periodic, so consider its Fourier series 1. The samples y n are a sufficient characterization of the central 2K+1 Fourier series coefficients. Use sampling theorem or graphically: Proof

SanDiego The Fourier series is a linear combination of K complex exponentials. These can be killed using a filter, which puts zeros at the location of the exponentials: This filter satisfies: because and is thus called annihilating filter. The filter has K+1 coefficients, but h 0 = 1. Thus, the K degrees of freedom {u k }, k = 1…K specify the locations. Proof

3. To find the coefficients of the annihilating filter, we need to solve a convolution equation, which leads to a K by K Toeplitz system 4. Given the coefficients {1, h 1, h 2, … h K }, we get the {t k } ’ s by factorization of where Proof

5. To find the coefficients {x k }, we have a linear problem, since given the {t k } ’ s or equivalently the {u k } ’ s, the Fourier series is given by This can be written as a Vandermonde system: which has always a solution for distinct {u k } ’ s. This completes the proof of uniqueness, using B   2K samples. Since B  is odd, we need 2K+1 samples, which is as close as it gets to Occam ’ s razor! Also, B > 2K/  = 

16 Notes on Proof The procedure is constructive, and leads to an algorithm: 1.Take 2K+1 samples y n from Dirichlet kernel output 2.Compute the DFT to obtain Fourier series coefficients -K..K 3.Solve Toeplitz system of size K by K to get H(z) 4.Find roots of H(z) by factorization, to get u k and t k 5.Solve Vandermonde system of size K by K to get x k. The complexity is: 1.Analog to digital converter 2.K Log K 3.K 2 4.K 3 (can be accelerated) 5.K 2 Or polynomial in K! Note 1: For size N vector, with K Diracs, O(K 3 ) complexity, noiseless Note 2: Method similar to sinusoidal retrieval in spectral estimation

17 What’s Maybe Surprising…. Bandlimited Manifold

18 Generalizations For the class of periodic FRI signals which includes – Sequences of Diracs – Non-uniform or free knot splines – Piecewise polynomials There are sampling schemes with sampling at the rate of innovation with perfect recovery and polynomial complexity Variations: finite length, 2D, local kernels etc

19 The Real, Noisy Case… Acquisition in the noisy case: where ‘’analog’’ noise is before acquisition (e.g. communication noise on a channel) and digital noise is due to acquisition (ADC, etc) Example: Ultrawide band (UWB) communication…. sampling kernel digital noiceanalog noice

20 too noisy? Annihilating Filter method Cadzow FTT Linear system yes no The Real, Noisy Case… Total Least Squares: Annihilation equation: can only be approximately satisfied. Instead: using SVD of a rectangular system Cadzow denoising: When very noisy: TLS can fail… Use longer filter L > K, but use fact that noiseless matrix A is Toeplitz of rank K Iterate between the two conditions Algorithm:

Resolution Resolution becomes a binary factor Necessity to identify a posteriori the regime of operation 21 Resolved Unresolved

1. Sparse Sampling Classic versus new results: Known versus unknown locations Finite rate of innovation and performance bounds The price of discretization 2. Sampling physics: Wave equation Sampling physical fields resulting from PDEs The plenacoustic function and sampling wave fields Wavefield synthesis Sampling the plenoptic field 3. Sampling fields: Trajectory sampling Spatial fields and mobile sampling Sampling along trajectories 5. Conclusions Outline 22

23 Sampling Versus Sampling Physics

24 Sampling and Interpolating Acoustic Fields Wave fields governed by the wave equation – Space-time distribution – Constrained by wave equation – Not arbitrary, but smoothed What can we say about sampling/interpolation? – Spatio-temporal Nyquist rate – Perfect reconstruction – Aliasing – Space-time processing How do we sample and interpolate? Wave equation Analog sources Space-time acoustic wave field

25 Many Microphones and Loudspeakers Multiple microphones/loudspeakers – physical world (e.g. free field, room) – distributed signal acquisition of sound with “many” microphones – sound rendering with many loudspeakers (wavefield synthesis) This is for real! – sound recording – special effects – movie theaters Questions: – “few” microphones and hear any location? – Dual question for synthesis (moving sources) – Implication on acoustic localization problems Method: – Time-Space Green function – Plenacoustic function MIT1020 mics

26 The Plenacoustic Function and its Sampling PAF: free field versus in a room for a given point source We plot: p(x,t), that is, the spatio-temporal impulse response The key question for sampling is:, that is, the Fourier transform A precise characterization of for large and will allow sampling and reconstruction error analysis Free field: easy: delay is proportional to (x 2 +d 2 ) 1/2

27 The Plenacoustic Function in Fourier Space  : temporal frequency  : spatial frequency

28 A Sampling Theorem for Acoustic Fields Theorem [AjdlerSV:06]: Assume a max temporal frequency Pick a spatial sampling frequency Spatio-temporal signal interpolated from samples taken at Argument: Take a cut through PAF Use exp. decay away from central triangle to bound aliasing Improvement using quincunx lattice

29 Visual Proof ;)

30 Application: Sound Field Reproduction Use the wave equation as an interpolation kernel! Control points at spatial Nyquist rate MIMO inversion or equalization The wave field is reproduced accurately in an entire region! Desired acoustic sceneReproduced acoustic scene

31 Application: Sound Field Reproduction Example Sinusoids Diracs

32 Plenoptic Function 101 – All the images you ever wanted to see and were afraid to ask! – Adelson 1990: f(x,y,z, , ,,t)! – How to sample it? – Also known as light field and image based rendering The Plenoptic Function and its Sampling

33 Sampling light fields [Stanford multi-camera array] 3D 2D 4D 5D [Imperial College multi-camera array]

34

35

36 Plenoptic Function in Fourier space – Collection of lines within a cone – Lines image space lines in Fourier space – Cone in Fourier space – Approximately bandlimited – Can be sampled – Camera spacing depends on depth of field

37 The Non-Bandlimitedness of the Plenoptic Function (MD, DMM) – a simple model – no luck, no bandlimit, only approximate bandlimit! – See M. N. Do et al, On the Bandwidth of the Plenoptic Function, in IEEE Transactions on Image Processing, 2012.M. N. DoOn the Bandwidth of the Plenoptic Function

38 The Non-Bandlimitedness of the Plenoptic Function (MD, DMM) – Analysis – Plane surface: scaling property of the Fourier transform – ‘’curved’’ surface: akin to modulation

1. Sparse Sampling Classic versus new results: Known versus unknown locations Finite rate of innovation and performance bounds The price of discretization 2. Sampling physics: Wave equation Sampling physical fields resulting from PDEs The plenacoustic function and sampling wave fields Wavefield synthesis Sampling the plenoptic field 3. Sampling fields: Trajectory sampling Spatial fields and mobile sampling Sampling along trajectories 5. Conclusions Outline 39

Trajectories 40

Key Problems in radioactivity measurements 1.Spatial estimation 2.Inverse problems: So what is the government not telling you? 3.Tampering/Outliers Detection – Public network: fake samples to hide possible leaks – Sensor reliability: generation of outliers 4.New Leaks Detection – Detect new releases using as few measurements as possible 41 ?!?

Famous historical example Chernobyl 42

43 On going projects in “Citizen Sensing” Citizen sensing – Pollution – Radioactivity Many open problems – Mobile sensing – Data quality – Safety Public projects – Safecast in Japan – Pollution monitoring

44 Pollution monitoring Pollution monitoring of NO2, NYC

45 Temperature monitoring Example on EPFL campus

46 2D Sampling theory Requires lattices

47 2D Sampling theory Sampling along trajectories Mobile sensor can sample along trajectory ‘’for free’’

48 2D Sampling theory: Trajectories

49 Conclusions: Sampling is Alive and Well! Physical problem and questions – Sparsity? – Low dimensional approximation? Modeling – PDE, BL/NBL – Measurement process Computational solution – FRI, CS: from the tool box – Conditioning Results – Reconstruction of physical field – Inverse problem Experimental set up – Lab set up – Real world data

Acknowledgements Sponsors and supporters – The conference organizers! – NSF Switzerland – ERC Europe: SPARSAM Co-authors and collaborators – Thierry Blu, CUHK – Ivana Jovanovic, SonoView – Yue Lu, Harvard – Yann Barbotin, EPFL – Ivan Dokmanic, EPFL – Mihailo Kolundzija, EPFL – Marta Martinez-Camara, EPFL – Juri Ranieri, EPFL Discussions and Interactions – V. Goyal, MIT – M. Unser, EPFL 50