1. Compressive Sampling (of Analog Signals) Moshe Mishali Yonina C. Eldar Technion – Israel Institute of Technology http://www.technion.ac.il/~moshikomoshiko@tx.technion.ac.il http://www.ee.technion.ac.il/people/YoninaEldar yonina@ee.technion.ac.il Advanced topics in sampling (Course 049029) Seminar talk – N ovember 2008

2. 2 Context - Sampling Digital worldAnalog world Continuous signal Reconstruction D2A Sampling A2D

3. 3 Compression Original 2500 KB 100% Compressed 950 KB 38% Compressed 392 KB 15% Compressed 148 KB 6% “Can we not just directly measure the part that will not end up being thrown away ?” Donoho

4. 4 Outline Mathematical background From discrete to analog Uncertainty principles for analog signals Discussion

5. 5 References M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors," IEEE Trans. on Signal Processing, vol. 56, no. 10, pp. 4692-4702, Oct. 2008. M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals," CCIT Report #639, Sep. 2007, EE Dept., Technion. Y. C. Eldar, "Compressed Sensing of Analog Signals", submitted to IEEE Trans. on Signal Processing, June 2008. Y. C. Eldar and M. Mishali, "Robust Recovery of Signals From a Union of Subspaces", arXiv.org 0807.4581, submitted to IEEE Trans. Inform. Theory, July 2008. # Y. C. Eldar, "Uncertainty Relations for Analog Signals", submitted to IEEE Trans. Inform. Theory, Sept. 2008.

6. 6 Mathematical background Basic ideas of compressed sensing Single measurement model (SMV) Multiple- and Infinite- measurement models (MMV, IMV) The “Continuous to finite” block (CTF)

7. 7 Compressed Sensing “Can we not just directly measure the part that will not end up being thrown away ?” Donoho “sensing … as a way of extracting information about an object from a small number of randomly selected observations” Candès et. al. Nyquist rate Sampling Analog Audio Signal Compression (e.g. MP3) High-rateLow-rate Compressed Sensing

8. 8 Concept Goal: Identify the bucket with fake coins. Nyquist: Weigh a coin from each bucket Compression Bucket # numbers1 number Compressed Sensing: Bucket # 1 number Weigh a linear combination of coins from all buckets

9. 9 Mathematical Tools non-zero entries  at least measurements Recovery: brute-force, convex optimization, greedy algorithms, and more…

10. 10 CS theory – on 2 slides Compressed sensing (2003/4 and on) – Main results Maximal cardinality of linearly independent column subsets Hard to compute ! is uniquely determined by Donoho and Elad, 2003

11. 11 is uniquely determined by is random with high probability Donoho, 2006 and Candès et. al., 2006 NP-hard Convex and tractable Greedy algorithms: OMP, FOCUSS, etc. Donoho, 2006 and Candès et. al., 2006 Tropp, Cotter et. al. Chen et. al. and many other Compressed sensing (2003/4 and on) – Main results CS theory – on 2 slides Donoho and Elad, 2003

12. 12 IMV = Infinite Measurement Vectors (countable or uncountable) with joint sparsity prior How can be found ? Sparsity models measurements unknowns SMV MMV Joint sparsity Infinite many variables Exploit prior  Reduce problem dimensions Infinite many constraints

13. 13 Reduction Framework Find a frame for Solve MMV Mishali and Eldar (2008) Theorem IMVMMV Deterministic reduction Infinite structure allows CS for analog signals

14. 14 From discrete to analog Naïve extension The basic ingredients of sampling theorem Sparse multiband model Rate requirements Multicoset sampling and unique representation Practical recovery with the CTF block Sparse union of shift-invariant model Design of sampling operator Reconstruction algorithm

15. 15 Naïve Extension to Analog Domain Standard CS Discrete Framework Analog Domain Sparsity prior what is a sparse analog signal ? Generalized sampling Finite dimensional elements Infinite sequence Continuous signal Operator Random is stable w.h.p Stability Randomness  Infinitely many Need structure for efficient implementation Finite program, well-studied Undefined program over a continuous signal Reconstruction

16. 16 Random is stable w.h.p Naïve Extension to Analog Domain Sparsity prior what is a sparse analog signal ? Generalized sampling Finite dimensional elements Infinite sequence Continuous signal Operator Stability Randomness  Infinitely many Need structure for efficient implementation Finite program, well-studied Undefined program over a continuous signal Reconstruction Questions: 1.What is the definition of analog sparsity ? 2.How to select a sampling operator ? 3.Can we introduce stucture in sampling and still preserve stability ? 4.How to solve infinite dimensional recovery problems ? Standard CS Discrete Framework Analog Domain

17. 17 A step backward Nyquist 1928 Shannon 1949 Kotelnikov 1933 “Success has many fathers …” Whittaker 1915 Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than

18. 18 A step backward Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than A signal model A minimal rate requirement Explicit sampling and reconstruction stages Fundamental ingredients of a sampling theorm

19. 19 Discrete Compressed Sensing Analog Compressive Sampling

20. 20 Analog Compressed Sensing A signal with a multiband structure in some basis no more than N bands, max width B, bandlimited to (Mishali and Eldar 2007) 1.Each band has an uncountable number of non-zero elements 2.Band locations lie on an infinite grid 3.Band locations are unknown in advance What is the definition of analog sparsity ? (Eldar 2008) More generally only sequences are non-zero

21. 21 Multi-Band Sensing: Goals bands SamplingReconstruction Goal: Perfect reconstruction Constraints: 1.Minimal sampling rate 2.Fully blind system AnalogInfiniteAnalog What is the minimal rate ? What is the sensing mechanism ? How to reconstruct from infinite sequences ?

22. 22 Rate Requirement Average sampling rate Theorem (non-blind recovery) Subspace scenarios: Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) Half blind system (Herley and Wong 99, Venkataramani and Bresler 00) Landau (1967) Theorem (blind recovery) Mishali and Eldar (2007) 1.The minimal rate is doubled. 2.For, the rate requirement is samples/sec (on average).

23. 23 Sampling Analog signal In each block of samples, only are kept, as described by Point-wise samples 0 2 3 0 0 2 2 3 3 Multi-Coset: Periodic Non-uniform on the Nyquist grid Bresler et. al. (96,98,00,01)

24. 24 The Sampler DTFT of sampling sequences Constant matrix known in vector form unknowns Length. known Problems: 1.Undetermined system – non unique solution 2.Continuous set of linear systems is jointly sparse and unique under appropriate parameter selection ( ) is sparse Observation:

25. 25 Paradigm Solve finite problem Reconstruct 0 1 2 3 4 5 6 S = non-zero rows

26. 26 CTF block Solve finite problem Reconstruct MMV Continuous to Finite ContinuousFinite span a finite space Any basis preserves the sparsity

27. 27 Algorithm CTF Continuous-to-finite block: Compressed sensing for analog signals Perfect reconstruction at minimal rate Blind system: band locations are unkown Can be applied to CS of general analog signals Works with other sampling techniques

28. 28 Blind reconstruction flow Spectrum-blind Sampling No Yes Spectrum-blind Reconstruction Uniform at Multi-coset with Universal Ideal low-pass filter SBR4 CTF SBR2 CTFBi-section Yes No

29. 29 Bresler et. al. (96,00) Final reconstruction (non-blind)

30. 30 Framework: Analog Compressed Sensing Sampling signals from a union of shift-invariant spaces (SI) generators Subspace

31. 31 Framework: Analog Compressed Sensing What happen if only K<<N sequences are not zero ? There is no prior knowledge on the exact indices in the sum Not a subspace ! Only k sequences are non-zero

32. 32 Framework: Analog Compressed Sensing Only k sequences are non-zero CTF Step 1: Compress the sampling sequences Step 2: “Push” all operators to analog domain System A High sampling rate = m/T Post-compression

33. 33 Framework: Analog Compressed Sensing CTF Eldar (2008) Theorem System B Low sampling rate = p/T Pre-compression

34. 34 Does it work ?

35. 35 Simulations Brute-ForceM-OMP Sampling rate Minimal rate

36. 36 Simulations (2) SBR4SBR2 Empirical recovery rate Sampling rate 0% Recovery100% Recovery0% Recovery100% Recovery Noise-free

37. 37 Simulations (3) SignalReconstruction filter Output Time (nSecs) Amplitude

38. 38 Break (10 min. please)

39. 39 Uncertainty principles Coherence and the discrete uncertainty principle Analog coherence and principles Achieving the lower coherence bound Uncertainty principles and sparse representations

40. 40 The discrete uncertainty principle Uncertainty principle

41. 41 Discrete coherence Which bases achieve the lowest coherence ?

42. 42 Discrete coherence Which signal achieves the uncertainty bound ? SpikesFourier

43. 43 Discrete to analog Shift invariant spaces Sparse representations Questions: What is the analog uncertainty principle ? Which bases has the lowest coherence ? Which signal achieves the lower uncertainty bound ?

44. 44 Analog uncertainty principle Eldar (2008) Theorem Eldar (2008) Theorem

45. 45 Bases with minimal coherence In the DFT domain Spikes Fourier What are the analog counterparts ? Constant magnitude Modulation “Single” component Shifts

46. 46 Bases with minimal coherence In the frequency domain

47. 47 Tightness

48. 48 Sparse representations In discrete setting

49. 49 Sparse representations Analog counterparts Undefined program ! But, can be transformed into an IMV model

50. 50 Discussion IMV model as a fundamental tool for treating sparse analog signals Should quantify the DSP complexity of the CTF block Compare approach with the “analog” model Building blocks of analog CS framework.

51. 51 Thank you