Yonina Eldar Department of Electrical Engineering Technion-Israel Institute of Technology Beyond Bandlimited Sampling: Nonideal Sampling, Smoothness and Sparsity Rice, April 2008

2 Sampling: “Analog Girl in a Digital World…” Judy Gorman 99 Digital worldAnalog world Signal processing Denoising Image analysis … Reconstruction D2A Sampling A2D (Interpolation)

3 Main Problem Can we reconstruct x(t) from c[n] ? No! Unless we know something about x(t) x(t) bandlimited x(t) piece-wise linear Different priors lead to different reconstructions Introduction Input Signals NonlinearitiesReconstruction

4 Our Point-Of-View The field of sampling was traditionally associated with methods implemented either in the frequency domain, or in the time domain Sampling can be viewed in a broader sense of projection onto any subspace or union of subspaces Can choose the subspaces to yield interesting new possibilities (below Nyquist sampling of sparse signals, pointwise samples of non bandlimited signals, perfect compensation of nonlinear effects …) Introduction Input Signals NonlinearitiesReconstruction

5 Bandlimited Sampling Theorems Cauchy (1841): Whittaker (1915) - Shannon (1948): Extensions focusing primarily on bandlimited signals with nonuniform grids A. J. Jerri, “The Shannon sampling theorem - its various extensions and applications: A tutorial review”, Proc. IEEE, pp , Nov Periodic & N-BL π -BL Introduction Input Signals NonlinearitiesReconstruction

6 Impractical Limitations of standard sampling theorems: Input bandlimited Ideal sampling Ideal reconstruction (ideal LPF) Towards More Robust DSPs … Towards more robust DSPs: General inputs Nonideal sampling: general pre-filters, nonlinear distortions Simple interpolation kernels Introduction Input Signals NonlinearitiesReconstruction

7 Two key ideas in bandlimited sampling: Avoid aliasing Fourier domain analysis Beyond Bandlimited Misleading concepts! Suppose that with Signal is clearly not bandlimited Aliasing in frequency and time Perfect reconstruction possible from samples Introduction Input Signals NonlinearitiesReconstruction Aliasing is not the issue …

8 Nonideal sampling Replace Fourier analysis by functional analysis, Hilbert space algebra, and convex optimization Fourier Domain Can Be Misleading Nonlinear distortion t=n linear distortion Original + Initial guess Reconstructed signal Introduction Input Signals NonlinearitiesReconstruction

9 Moving To An Abstract Hilbert Space Subspace prioir Shift invariant subspace: General subspace in a Hilbert space Smoothness constraints: Sparse vector model (compressed sensing): Bandlimited Spline spaces Analog Version? What classes of inputs would we like to treat? Introduction Input Signals NonlinearitiesReconstruction

10 Broad Sampling Framework Broad class of input signals: Subspace priors, smoothness priors Sparse analog signals: Signals restricted to bands General pre-filters Perfect compensation of nonlinear distortions Nonideal reconstruction filters Beyond Ideal Bandlimited Sampling Analog sampling + compressed sensing Introduction Input Signals NonlinearitiesReconstruction

11 Outline 1. Input Signals: Subspace methods: Perfect recovery from linear generalized samples Smoothness priors: Minimax approximations Sparsity priors: Brief overview of compressed sensing (CS) for finite vectors CS of analog signals: Blind sampling of multiband signals 2. Nonlinearties: Perfect recovery in the presence of nonlinear distortions 3. Nonideal reconstruction: Minimax approximation with simple kernels Introduction Input Signals NonlinearitiesReconstruction

12 Non-Ideal Linear Sampling Non- ideal sampling (Riesz: Any linear and bounded acquisition) Examples: Sampling space S: In the sequel: T=1 s(-t) t=nT Generalized anti- aliasing filter Electrical circuit 0 Δ Local averaging Sampling functions Introduction Input Signals NonlinearitiesReconstruction

13 Perfect Reconstruction Key observation: Knowing is equivalent to knowing if is a frame (Riesz basis): If x(t) is in S then and perfect reconstruction is possible Given s(t) which signals can be perfectly reconstructed? sampling space Introduction Input Signals NonlinearitiesReconstruction

14 Shannon Revisited Perfect reconstruction scheme: Bandlimited sampling: Introduction Input Signals NonlinearitiesReconstruction

15 Mismatched Sampling What if x(t) lies in a subspace where A is generated by a(t) ? If then PR impossible since If then PR possible (Christansen and Eldar, 2005) Perfect Reconstruction in a Subspace: Introduction Input Signals NonlinearitiesReconstruction

16 Examples Bandlimited sampling: Can x(t) be recovered even though it is not bandlimited? Point-wise sampling of : c[n]=x[n] corresponding to s(t)=δ(t) Can recover x(t) as long as (Unser and Aldroubi 94) Introduction Input Signals NonlinearitiesReconstruction

17 Perfect Recovery 1. Compute convolutional inverse of 2. Convolve the samples with 3. Reconstruct with Introduction Input Signals NonlinearitiesReconstruction

18 Summary: Perfect Recovery In A Subspace General input signals (not necessarily BL) General samples (anti-aliasing filters) Results hold also for nonuniform sampling and more general spaces Introduction Input Signals NonlinearitiesReconstruction

19 No subspace information but Many consistent solutions Motivation: Want to be close to x(t) Minimize the worst-case difference: Complicated problem but … simple solution Smoothness Prior (Eldar 2007) optimal interpolation kernel Introduction Input Signals NonlinearitiesReconstruction

20 From Smoothness to Compressed Sensing Non ideal sampling of analog signals PR with subspace prior Approximations with smoothness prior Sparsity prior : Discrete signals If every 2K columns of A are linearly independent then there is a unique K-sparse signal (Donoho and Elad 03) Key observation: c can be relatively short and still contain the entire information about x Sparse prior: Samples: Can x be reconstructed from c? Introduction Input Signals NonlinearitiesReconstruction

21 Joint Sparsity Multiple measurement vectors (MMV): C=AX Each column of X is K-sparse The non-zero values share a common location set (Chen and Huo 06, Mishali and Eldar 07) Introduction Input Signals NonlinearitiesReconstruction Theorem Let X have. If then X is the unique sparsest solution set

22 Algorithms SMV Efficient algorithms: Basis pursuit Matching pursuit Others To overcome NP-hard MMV Efficient algorithms: M-Basis pursuit M-Matching pursuit Others Introduction Input Signals NonlinearitiesReconstruction Results and Algorithms Inherently Discrete

23 Analog Compressed Sensing Introduction Input Signals NonlinearitiesReconstruction What is analog compressed sensing? A signal with a multiband structure in some basis Previous methods for analog CS involve discretization or finite models (R. Baraniuk, J. Laska, S. Kirolos, M. Duarte, T. Ragheb, Y. Massoud, A. Gilbert, M. Iwen, M. Strauss, J. Tropp, M. Wakin, D. Baron) Our model is inherently continuous: each band has an uncountable number of non-zero elements No finite basis! no more than N bands, max width B, bandlimited to

24 Goals Minimal rate Perfect reconstruction Reconstruction Sampling Blind Reconstruction Blind Sampling Blind system 123 Introduction Input Signals NonlinearitiesReconstruction

25 Non-Blind Scenario Average sampling rate Theorem Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) Subspace scenario Half blind system (Herley and Wong 99, Venkataramani and Bresler 00) Landau (1967) is constant with Lebesgue measure Introduction Input Signals NonlinearitiesReconstruction

26 Minimal Sampling Rate Question: What is the minimal sampling rate that allows blind perfect reconstruction with arbitrary sampling/reconstruction methods? Minimal sampling rate for our set M: 2NB Theorem Mishali and Eldar (2007) The minimal sampling rate is doubled (for ) Introduction Input Signals NonlinearitiesReconstruction

27 Sampling Analog signal In each block of samples, only are kept, as described by Point-wise samples Multi-Coset: Periodic Non-uniform on the Nyquist grid Introduction Input Signals NonlinearitiesReconstruction

28 The Sampler DTFT of sampling sequences Constant matrix known in vector form unknowns Length. known Introduction Input Signals NonlinearitiesReconstruction

29 Reconstruction Objectives Problems: Goal: Recover 1.Undetermined system – non unique solution (p<L) 2.Continuous set of linear systems Observation: is sparse Introduction Input Signals NonlinearitiesReconstruction

30 Uniqueness Theorem (Mishali and Eldar, 2007)  Introduction Input Signals NonlinearitiesReconstruction

31 Choose a dense grid of Solve for each Interpolate Disadvantages: Loose perfect reconstruction Large computational complexity Sensitivity to noise  We Say NO to Discretization ! Introduction Input Signals NonlinearitiesReconstruction

32 Paradigm Solve finite problem Reconstruct Introduction Input Signals NonlinearitiesReconstruction

33 Once S is Known… Solve finite problem Reconstruct Reconstruct exactly by Introduction Input Signals NonlinearitiesReconstruction

34 CTF block Solve finite problem Reconstruct MMV Continuous to Finite ContinuousFinite Introduction Input Signals NonlinearitiesReconstruction

35 CTF Fundamental Theorem Mishali and Eldar (2007) Theorem Introduction Input Signals NonlinearitiesReconstruction

36 Algorithm CTF Introduction Input Signals NonlinearitiesReconstruction Continuous-to-finite block: Compressed sensing for analog signals Perfect reconstruction at minimal rate Blind system: band locations are unkown

37 Summary: Perfect Reconstruction Perfect reconstruction from subspace samples, sparse samples Minimax reconstruction from smooth signals Linear sampling Ideal interpolation kernels Perfect compensation for nonlinear distortions Minimax interpolation with simple kernels Until Now: Limitations: Coming up …. Introduction Input Signals NonlinearitiesReconstruction

38 Nonlinear Sampling Saturation in CCD sensors Dynamic range correction Optical devices High power amplifiers Many applications… No theory! s(-t) Memoryless nonlinear distortion t=n Introduction Input Signals NonlinearitiesReconstruction

39 Perfect Reconstruction Setting: m(t) is invertible with bounded derivative y(t) is lies in a subspace A Uniqueness same as in linear case! Proof: Based on extended frame perturbation theory and geometrical ideas If and m(t) is invertible and smooth enough then y(t) can be recovered exactly ( Dvorkind, Eldar, Matusiak 2007) If and m(t) is invertible and smooth enough then y(t) can be recovered exactly ( Dvorkind, Eldar, Matusiak 2007) Theorem (uniqueness): Introduction Input Signals NonlinearitiesReconstruction

40 Algorithm: Linearization Transform the problem into a series of linear problems: 1.Initial guess y 0 2.Linearization: Replace m(t) by its derivative around y 0 3.Solve linear problem and update solution y n y n+1 Introduction Input Signals NonlinearitiesReconstruction error in samples correction solving linear problem Questions: 1. Does the algorithm converge? 2. Does it converge to the true input?

41 Main idea: 1. Minimize error in samples where 2. From uniqueness if Perfect reconstruction global minimum of Difficulties: 1. Nonlinear, nonconvex problem 2. Defined over an infinite space Optimization Based Approach Under the previous conditions any stationary point of is unique and globally optimal ( Dvorkind, Eldar, Matusiak 2007) Under the previous conditions any stationary point of is unique and globally optimal ( Dvorkind, Eldar, Matusiak 2007) Theorem : Introduction Input Signals NonlinearitiesReconstruction Our algorithm traps a stationary point!

42 Simulation Example Optical sampling system: optical modulator ADC Introduction Input Signals NonlinearitiesReconstruction

43 Simulation Initialization with First iteration: Third iteration: Introduction Input Signals NonlinearitiesReconstruction

44 Constrained Reconstruction Consistent reconstruction (Unser and Aldroubi 94, Eldar 03,04) Unique solution possible only if: Problem: Resulting error can be quite large Introduction Input Signals NonlinearitiesReconstruction reconstruction space

45 Minimax Reconstruction Motivation: Want to be close to x(t) Best approximation in W is but can’t be attained from Minimize the worst-case difference: Complicated problem but …. Simple solution: Comparison: (Eldar and Dvorkind, 2005) Introduction Input Signals NonlinearitiesReconstruction

46 Example: Audio Processing LPF 1 /2 LPF 2 x2 H 8[kHz] Down-up sampling with non-ideal filtering: Original signal No processing (NE=0.81) Consistent (NE=0.87) Regret (NE=0.28) Orthogonal Projection (NE=0.27) Introduction Input Signals NonlinearitiesReconstruction

47 Conclusion Broad class of input signals: Subspace priors, smoothness priors Compressed sensing for analog signals Compensations for many practical distortions Applicable to a wide host of sampling problems Beyond Ideal Bandlimited Sampling Can beat Nyquist and aliasing using the right tools!

48 References Y. C. Eldar and T. Dvorkind, "A Minimum Squared-Error Framework for Generalized Sampling," IEEE Trans. Signal Processing, vol. 54, no. 6, pp , June M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals,“ submitted to IEEE Trans. on Signal Processing, Sep T. G. Dvorkind, Y. C. Eldar and E. Matusiak, "Nonlinear and Non-Ideal Sampling: Theory and Methods," submitted to IEEE Trans. on Signal Processing, Nov M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors", submitted to IEEE Trans. on Signal Processing, Feb Y. C. Eldar and M. Unser,"Nonideal Sampling and Interpolation from Noisy Observations in Shift-Invariant Spaces," IEEE Trans. Signal Processing, Vol. 54, No. 7, pp , July Y. C. Eldar, "Sampling and Reconstruction in Arbitrary Spaces and Oblique Dual Frame Vectors ", J. Fourier Analys. Appl., vol. 1, no. 9, pp , Jan O. Christensen and Y. C. Eldar, "Oblique Dual Frames and Shift-Invariant Spaces," Applied and Computational Harmonic Analysis, vol. 17/1, pp , July 2004.

49 Details: where : Maximal angle between the spaces y x m’(t)