1. Beyond Nyquist: Compressed Sensing of Analog Signals
Yonina Eldar
Technion – Israel Institute of Technology
Read title
This is a joint work with my supervisor Prof. Yonina Eldar
Dagstuhl Seminar
December, 2008

2. Sampling: “Analog Girl in a Digital World…” Judy Gorman 99
Analog world
Digital world
Sampling
A2D
Signal processing
Denoising
Image analysis…
Reconstruction
D2A
We are used to point a file on our computer and say this is a picture
We used to listen to our digital music collection.
This is abuse of notation. We don’t really listen or watch the digital files. We see a conversion to analog signal of this digitally-stored information.
Let’s review the cycle of analog to digital.
This front-end of everything we know is so “transparent” to us that we are used to skip it. BUT a major research is done about efficient implementation of these stages.
(Interpolation) 2

3. Compression Compressed 148 KB 6% Compressed 950 KB 38%
“Can we not just directly measure the part that will not end up being thrown away ?”
Donoho
Compressed KB 6%
Compressed KB 38%
Compressed KB 15%
Original KB 100%

4. Outline Can break the Shannon-Nyquist barrier
Compressed sensing – background
From discrete to analog
Goals
Part I : Blind multi-band reconstruction
Part II : Analog CS framework
Implementations
Uncertainty relations
Can break the Shannon-Nyquist barrier
by exploiting signal structure

5. CS Setup K non-zero entries  at least 2K measurements
To demonstrate this idea CS was initially demonstrated in discrete framework.. We have a set of numebr (=buckets). Some of them contain values while the other are empty.
Goal find the locations and values of the non-zero elements, without directly measuring each of the entries of x
Way : mix in different ways
We do expect 2k values since we can define x exactly by a set of 2k numbers… however we don’t want to measure each of the elements in order to get this info. We want to directly acquire those 2k numbers…
But when we have less equations than unknowns we must incorporate a prior in order to solve the system
Emphasize recovery methods are also discrete
Recovery: brute-force, convex optimization, greedy algorithms, … 5

6. Brief Introduction to CS
Uniqueness:
is uniquely determined by
Donoho and Elad, 2003
is random
with high probability
Donoho, 2006 and Candès et. al., 2006
Recovery:
אני לא אוכל לסכם הכל בכמה שקפים אבל אתן רק את נקודות המפתח
הספרות בנושא מתעסקת בעיקר בשני התחומים האלה, כלומר איזה מטריצות, באיזה גודל, ובאיזה הסתברות הם "טובות". ונראה בהמשך הגדרות טובות יותר מקרוסקל-רנק.
כמו כן, שיטות מהירות לפתרון הבעיה, ובאיזה מידה הם מייצרות את הפתרון ה- sparsest ?
Convex and tractable
Donoho, 2006 and Candès et. al., 2006
NP-hard
Greedy algorithms: OMP, FOCUSS, etc.
Tropp, Elad, Cotter et. al,. Chen et. al., and many others

7. Naïve Extension to Analog Domain
Standard CS
Discrete Framework
Analog Domain
Sparsity prior
what is a sparse analog signal ?
Generalized sampling
Infinite
sequence
Operator
Continuous signal
Finite dimensional elements
Stability
Random is stable w.h.p
Randomness  Infinitely many
Need structure for efficient implementation
Reconstruction
Finite program, well-studied
Undefined program over a continuous signal 7

8. Naïve Extension to Analog Domain
Standard CS
Discrete Framework
Analog Domain
Questions:
What is the definition of analog sparsity ?
How to select a sampling operator ?
Can we introduce stucture in sampling and still preserve stability ?
How to solve infinite dimensional recovery problems ?
Sparsity prior
what is a sparse analog signal ?
Generalized sampling
Infinite
sequence
Operator
Continuous signal
Finite dimensional elements
Stability
Random is stable w.h.p
Randomness  Infinitely many
Need structure for efficient implementation
Reconstruction
Finite program, well-studied
Undefined program over a continuous signal 8

9. Goals Concrete analog sparsity model Reduce sampling rate (to minimal)
Simple recovery algorithms
Practical implementation in hardware

10. What is the definition of analog sparsity ?
Analog Compressed Sensing
What is the definition of analog sparsity ?
A signal with a multiband structure in some basis
Each band has an uncountable number of non-zero elements
Band locations lie on an infinite grid
Band locations are unknown in advance
no more than N bands, max width B, bandlimited to
(Mishali and Eldar 2007)
More generally
only sequences are non-zero (Eldar 2008)

11. Multiband “Sensing” bands
(Mishali and Eldar 2007)
bands
Sampling
Reconstruction
Analog
Infinite
Analog
Goal: Perfect reconstruction
Known band locations (subspace prior):
Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98)
Half blind system (Herley and Wong 99, Venkataramani and Bresler 00)
We are interested in unknown spectral support (a union of subspace prior)
Next steps:
What is the minimal rate requirement ?
A fully-blind system design 11

12. Rate Requirements Theorem (non-blind recovery)
Landau (1967)
Average sampling rate
The minimal rate is doubled
For , the rate requirement is samples/sec (on average)
Theorem (blind recovery)
Mishali and Eldar (2007)

13. Sampling Multi-Coset: Periodic Non-uniform on the Nyquist grid
In each block of samples, only are kept, as described by 2
Analog signal
Point-wise samples 3 3 2 3 2 13

14. The Sampler in vector form unknowns DTFT of sampling sequences
Constant
is sparse
Observation:
Length .
known
matrix
known
Problems:
Undetermined system – non unique solution
Continuous set of linear systems
is jointly sparse and unique under appropriate parameter selection ( )

15. Paradigm Solve finite problem Reconstruct S = non-zero rows 1 2 3 4 5
S = non-zero rows 1 2 3 4 5 6 15

16. Continuous to Finite CTF block Continuous Finite MMV
Solve finite problem
Reconstruct
span a finite space
Any basis preserves the sparsity
Continuous
Finite

17. 2-Words on Solving MMV
Find a matrix U that has as few non-zero rows as possible
Variety of methods based on optimizing mixed column-row norms
We prove equivalence results by extending RIP and coherence to allow for structured sparsity (Mishali and Eldar, Eldar and Bolcskei)
New approach: ReMBo – Reduce MMV and Boost
Main idea: Merge columns of V to obtain a single vector problem y=Aa
Sparsity pattern of a is equal to that of U
Can boost performance by repeating the merging with different coeff.

18. Algorithm
CTF
Continuous-to-finite block: Compressed sensing for analog signals
Perfect reconstruction at minimal rate
Blind system: band locations are unknown
Can be applied to CS of general analog signals
Works with other sampling techniques

19. Framework: Analog Compressed Sensing
(Eldar 2008)
Sampling signals from a union of shift-invariant spaces (SI)
Subspace
generators 19

20. Framework: Analog Compressed Sensing
What happen if only K<<N sequences are not zero ?
Not a subspace !
There is no prior knowledge on the exact indices in the sum
Only k sequences are non-zero 20

21. Framework: Analog Compressed Sensing
Step 1: Compress the sampling sequences
Step 2: “Push” all operators to analog domain
CTF
System A
High sampling rate = m/T Post-compression
Only k sequences are non-zero 21

22. Framework: Analog Compressed Sensing
Low sampling rate = p/T Pre-compression
System B
CTF
Theorem
Eldar (2008) 22

23. Reconstruction filter
Simulations
Signal
Reconstruction filter
Amplitude
Amplitude
Time (nSecs)
Output
Finally, we tested our methods when only a truncated time-interval of the signal is available, and the reconstruction filters used to combine the slice (once S is known) are also non-ideal.
Time (nSecs)

24. Simulations Brute-Force M-OMP Minimal rate Minimal rate Sampling rate
We conducted simulations for N=3 bands with exact width of B=1.05 GHz out of total spectrum width of 20 ghz
We report the recovery rate over extensive set of test signals, and for different sampling rates.
Here we demonstrate reconstruction when the MMV is solved either with brute-force or the greedy algorithm multi-ortho. Matching pursuit. Other methods can be employed.
Sampling rate
Sampling rate
Brute-Force
M-OMP

25. Empirical recovery rate
Simulations
0% Recovery
100% Recovery
0% Recovery
100% Recovery
Noise-free
In addition, we tested our algorithms in the presence of noise.
This simulations uses M-Omp and as evident, the robustness of both algorithms, follows out from the robustness of the underlying MMV technique.
Sampling rate
Sampling rate
SBR4
SBR2
Empirical recovery rate

26. Multi-Coset Limitations
Analog signal 2
Point-wise samples 3 3 2 3 2
Delay
ADC
@ rate
Impossible to match rate for wideband RF signals (Nyquist rate > 200 MHz)
Resource waste for IF signals
3. Requires accurate time delays

27. Efficient implementation
Efficient Sampling
(Mishali, Eldar, Tropp 2008)
Efficient implementation
Use CTF

28. Hardware Implementation
A few first steps…

29. Pairs Of Bases Both and are small!
Until now: sparsity in a single basis
Can we have a sparse representation in two bases?
Motivation: A combination of bases can sometimes better represent the signal
Both and are small!

30. Uncertainty Relations
How sparse can be in each basis?
Finite setting: vector in
Elad and Brukstein 2002
Different bases
Uncertainty
relation

31. Analog Uncertainty Principle
Theorem
Eldar (2008)
Eldar (2008)
Theorem

32. Bases With Minimal Coherence
In the DFT domain
Spikes
Fourier
What are the analog counterparts ?
Constant magnitude
Modulation
“Single” component
Shifts

33. Analog Setting: Bandlimited Signals
Minimal coherence:
Tightness:

34. Finding Sparse Representations
Given a dictionary ,
expand using as few elements as possible:
minimize
Solution is possible using CTF if is small enough
Basic idea:
Sample with basis
Obtain an IMV model:
maximal value
Apply CTF to recover
Can establish equivalence with as long as is small enough

35. Conclusion Compressed Sensing of Analog Signals
Extend the basic results of CS to the analog setting - CTF
Sample analog signals at rates much lower than Nyquist
Can find a sparse analog representation
Can be implemented efficiently in hardware
Questions:
Other models of analog sparsity?
Other sampling devices?

36. Some Things Should Remain At The Nyquist Rate
Thank you
High-rate
Thank you
יש דברים שעדיין צריך להגיד בקצב נייקויסט

37. References
M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals,“ to appear in IEEE Trans. on Signal Processing.
M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors", IEEE Trans. on Signal Processing, vol. 56, no. 10, pp , Oct
Y. C. Eldar , "Compressed Sensing of Analog Signals", submitted to IEEE Trans. on Signal Processing.
Y. C. Eldar and M. Mishali, "Robust Recovery of Signals from a Union of Subspaces’’, submitted to IEEE Trans. on Inform. Theory.
Y. C. Eldar, "Uncertainty Relations for Analog Signals",  submitted to IEEE Trans. Inform. Theory.
Y. C. Eldar and T. Michaeli, "Beyond Bandlimited Sampling: Nonlinearities, Smoothness and Sparsity", to appear in IEEE Signal Proc. Magazine.