1. SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION * Pancham Shukla Communications and Signal Processing Group Imperial College London  This research is supported by EPSRC. Dr P L Dragotti by supervisor A transfer talk on 1/3/2005

2. 3 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION OUTLINE 1. INTRODUCTION Sampling: Problem, Background, and Scope 2. SIGNALS WITH FINITE RATE OF INNOVATION (FRI) (non-bandlimited) Definition, Extension in 2-D 2-D Sampling setup, Sampling kernels and their properties 3. SAMPLING OF FRI SIGNALS SETS OF 2-D DIRACS Local reconstruction (amplitude and position) BILEVEL POLYGONS & DIRACS using COMPLEX MOMENTS Global reconstruction (corner points) PLANAR POLYGONS using DIRECTIONAL DERIVATIVES Local reconstruction (corner points) 4. CONCLUSION AND FUTURE WORK

3. 4 Why sampling? Many natural phenomena are continuous (e.g. Speech, Remote sensing) and required to be observed and processed by sampling. Many times we need reconstruction (perfect !) of the original phenomena. Continuous  Discrete (samples)  Continuous Sampling theory by Shannon (Kotel’nikov, Whittaker) Why not always ‘bandlimited-sinc’? (Although powerful and widely used since 5 decades) 1. Real world signals are non-bandlimited. 2. Ideal low pass (anti-aliasing, reconstruction) filter does not exist. (Acquisition devices) 3. Shannon’s reconstruction formula is rarely used in practice with finite length signals (esp. images) due to infinite support and slow decay of ‘sinc’ kernel. (do we achieve PR in practice?) SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 1. INTRODUCTION ‘bandlimited-sinc’ scenario with the assumption of Perfect Reconstruction ! )(tx PR ? Motivation:

4. 5 Extensions of Shannon’s theory So… many papers but for comprehensive account, we refer to [Jerry 1977, Unser 2000]. Shift-invariant subspaces [Unser et al.] T he classes of non-bandlimited signals (e.g. uniform splines) residing in the shift-invariant subspaces can be perfectly reconstructed. The other non-bandlimited signals are approximated through their projections. SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 1. INTRODUCTION contd. We look into: Non-bandlimited signals that do not reside in shift-invariant subspace but have a parametric representation. Non-traditional ways of perfect reconstruction… ….from the projections of such signals in the shift- invariant subspace. Is it possible to perfectly reconstruct such signals from their samples? Any examples of such signals ? What type of kernels ? Sampling and reconstruction schemes? e.g. uniform spline Non-bandlimited signal projection Shift-invariant subspace

5. 6 Very recently such signals are identified and termed as Signals with Finite Rate of Innovation (or FRI signals) [ Vetterli et al. 2002] Model: Non-bandlimited signals that do not reside in shift-invariant subspace. Examples: Streams of Diracs, non-uniform splines, and piecewise polynomials. Unique feature: A finite number of degrees of freedom per time (rate of innovation  ) e.g. a Dirac in 1-D has a rate of innovation = 2 (i.e. amplitude and position). The sampling schemes for such signals in 1-D are given by [Vetterli, Marziliano and Blu 2002]. Extensions of these schemes in 2-D are given by [Maravic and Vetterli 2004], however, focusing on Sampling kernels: as sinc and Gaussian. Algorithms: Little more involved reconstruction algorithms (solution of linear systems, root finding) based on Annihilating filter method [from Spectral estimation, Error correction coding]. Reconstruction: Only a finite number of samples (   ) guarantees perfect reconstruction. SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 2. FRI SIGNALS

6. 7 Assortments of kernels [Dragotti, Vetterli and Blu, ICASSP-2005] For 1-D FRI signals, one can use varieties of kernels such as 1.That reproduce polynomials (satisfy Strang and Fix conditions) 2.Exponential Splines (E-Splines) [Unser] 3.Functions with rational Fourier transforms Our Focus Sampling extensions in 2-D using above mentioned kernels, in particular, for Sets of 2-D Diracs  Local & Global schemes: Local kernels & Complex moments + (AFM) Bilevel polygons  Global scheme: Complex moments + Annihilating filter method (AFM) Planar polygons  Local scheme: Directional derivatives + Directional kernels SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 2. FRI SIGNALS contd.

7. 8 Sampling setup Properties of sampling kernels In current context, any kernel that reproduce polynomials  of degrees  =0,1,2…  -1 such that Partition of unity: Reproduction of polynomials along x-axis: Reproduction of polynomials along y-axis: SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 2. SAMPLING FRI SIGNALS in 2-D e.g., B-Splines (biorthogonal) and Daubechies scaling functions (orthogonal) are valid kernels Input signal Sampling kernel Set of samples in 2-D

8. 9 Sets of 2-D Diracs: Local reconstruction Consider and with support such that there is at most one Dirac in an area of size. Assume. From the partition of unity (reproducing of polynomial of degree 0),it follows that SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 3. SETS OF 2-D DIRACS (property: partition of unity) This is derived as follows, B-Splines of order one The amplitude Only inner products overlap the unique Dirac

9. 10 Properties of sampling kernels In current context, any kernel that reproduce polynomials  of degrees  =0,1,2…  -1 such that Partition of unity: Reproduction of polynomials along x-axis: Reproduction of polynomials along y-axis: SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 2. SAMPLING FRI SIGNALS in 2-D e.g., B-Splines (biorthogonal) and Daubechies scaling functions (orthogonal) are valid kernels

10. 11 Sets of 2-D Diracs: Local reconstruction contd. … and using polynomial reproduction properties along x and y directions, the coordinate positions are given by SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 3. SETS OF 2-D DIRACS contd. Above relations are derived as, Similarly, it is easy to follow that As long as any two Diracs are sufficiently apart, we can accurately reconstruct a set of Diracs, considering one Dirac per time.

11. 12 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments Complex-moments for polygonal shapes: earlier works Since decades, moments are used to characterize unspecified objects. [Shohat and Tamarkin 1943, Elad et al. 2004]. Here, we present a sampling perspective to the results of [Davis 1964, Milanfar et al. 1995, Elad et al. 2004] on reconstruction of polygonal shapes using complex-moments. Milanfar et al. (1995 ) extended the above work using as follows Result of Davis (1964): For any non-degenerate, simply connected polygon with corner points ( ) in closure of any analytic function,following holds where are complex weights that depend on the ordered connection of corner points Definition: The nth simple and weighted complex-moments of a given function over a complex Cartesian plane in the closure are given by Simple momentWeighted moment

12. 13 Complex-moments for polygonal shapes: a modern connection SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments (simple moment) (weighted moment) Results of Milanfar et al. (1995): Milanfar et al. considered a bilevel polygon that is non-degenerate, simply connected and convex. They showed that when and is ‘1’ in the closure and ‘0’ out side. It follows that for Theorem [Milanfar et al.]: For a given non-degenerate, simply connected, and convex polygon in the complex Cartesian plane, all its N corner points are uniquely determined by its weighted complex- moments up to order 2N-1. Now, we will briefly review the annihilating filter method due to its relevance in finding weights and positions of the corner points z i from the observed complex moments.

13. 14 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments Annihilating filter method (we refer to [Vetterli, Stoica and Moses] for more details) This method is well known in the field of Error-correcting codes and Spectral estimation. Especially, in second application, it is employed to determine the weights and locations of the spectral components, generally observed in form of The annihilating filter method consists of the following steps: 1. Design the annihilating filter A(z): such that for filter with its z-transform the condition holds. 2. Locations: The convolution condition is solved by the following Yule-Walker system 3. Weights: Once the locations are known, eq. (1) is solved for the weights by the following Vandermonde system where (1) The roots of the filter are the locations Gives the weights

14. 15 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments A sampling perspective (using Complex-moments + Annihilating filter method) Consider g(x,y) as a non-degenerate, simply connected, and convex bilevel polygon with N corner points Consider g(x,y) as a set of N 2-D Diracs can reproduce polynomials up to degree 2N-1 (  = 0,1...2N-1) Then from the complex-moments formulation of Milanfar et al. Because of the polynomial reproduction property of the kernel, we derive that where are complex weights and are corner points of the bilevel polygon where denotes amplitudes of the Diracs and are complex position of the Diracs Now using annihilating filter method, it is straightforward to see that

15. 16 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments Simulation results Bilevel polygon with N=3 corner points A set of N=3 Diracs

16. 17 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 4. BILEVEL POLYGONS & DIRACS: Complex-moments Summary 1.A polygon with N corner points is uniquely determined from its samples using a kernel that reproduce polynomials up to degree 2N-1 along both x and y directions. 2.A set of Diracs is uniquely determined from its samples using a kernel that reproduces polynomials up to degree 2N-1 along both x and y directions and that there are at most N Diracs in any distinct area of size. 3.Global reconstruction algorithm Complexity  Complexity of the signal Numerical instabilities in algorithmic implementations for very close corner points

17. 18 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Problem formulation Intuitively, for a planar polygon, two successive directional derivatives along two adjacent sides of the polygon result into a 2-D Dirac at the corner point formed by the respective sides. Continuous model: Discrete challenge: Lattice theory: Directional derivatives  Discrete differences Subsampling over integer lattices and Local directional kernels in the framework of 2-D Dirac sampling (local reconstruction) In present context, we have access to samples only. Planar polygon with N corner points N pairs of orientations N pairs of directional derivatives N Diracs Local reconstruction scheme of 2-D Diracs Amplitudes and positions of the corner points

18. 19 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Lattice theory We refer to [Cassels, Convey and Sloan] for more detail. Base lattice: is a subset of points of Z 2 (integer lattice) Each pattern of subsampling (or ) over the integer lattice is characterized by a non-unique Sampling matrix For example, by taking, the base lattice is illustrated as A

19. 20 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Proposed sampling scheme Consider a planar polygon with corner points and a sampling kernel that satisfies partition of unity (reproduces polynomial of degree zero). The observed samples are given by Therefore, using lattice theory, apply a pair of directional differences and along and over the samples identified by the base lattice and its sampling matrix It then follows, By using Parseval’s identities and after certain manipulations, we have where Dirac at AModified kernel

20. 21 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives where Dirac at AModified kernel Directional kernels Modified kernel is a ‘directional kernel’. For each corner point  independent directional kernel. For example, support:

21. 22 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Local reconstruction of the corner point The directional kernel can reproduce polynomials of degrees 0 and 1 in both x and y directions. Assume that there is only one corner point  support of its associated directional kernel. Then from the local reconstruction scheme of Diracs, we can reconstruct the amplitude and the position of an equivalent Dirac at a given corner point (e.g. at point A ) as: A

22. 23 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Simulation result Original polygon with 3 corner points After first pair of directional difference on samples Samples of the polygon using Haar kernel After third pair of directional difference on samples After second pair of directional difference on samples Pair of directional differences Local reconstruction

23. 24 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 5. PLANAR POLYGONS: Directional-derivatives Planar polygon with N corner points N pairs of orientations N pairs of directional derivatives N Diracs Local reconstruction scheme of 2-D Diracs Amplitudes and positions of the corner points Summary: reconstruction algorithm Initial intuition:Final realization: Planar polygon with N corner points N pairs of orientations N pairs of directional differences N Diracs Local reconstruction scheme of 2-D Diracs Amplitudes and positions of the corner points Only one corner point in the support of its directional kernel Enough number of samples Advantage: 1.Local reconstruction. 2.Only local reconstruction complexity, irrespective of the number of corner points in a polygon.

24. 25 Conclusion We have proposed several sampling schemes for the classes of 2-D non-bandlimited signals. In particular, sets of Diracs and (bilevel and planar) polygons can be reconstructed from their samples by using kernels that reproduce polynomials. Combining the tools like annihilating filter method, complex-moments, and directional derivatives, we provide local and global sampling choices with varying degrees of complexity. SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 6. CONCLUSION & FUTURE WORK

25. 26 SAMPLING SCHEMES FOR 2-D SIGNALS WITH FINITE RATE OF INNOVATION 6. CONCLUSION & FUTURE WORK Future work From March 2005 to October 2005: 1.Exploring a different class of kernels, namely, exponential splines (E-Splines). 2.Extending the sampling schemes in higher dimension. For instance, using the notion of complex numbers in 4-D (quaternion). 3.Considering more intricate cases such as piecewise polynomials inside the polygons, and planar shapes with piecewise polynomial boundaries. We plan to submit a paper for IEEE Transactions on Image Processing by summer 2005. From November 2005 to June 2006: 1.Studying the wavelet footprints [Dragotti 2003] and then extending them in 2-D 2.Integrating the proposed sampling schemes with the footprints in 2-D 3.Investigating the sampling situations when the signals are perturbed with the noise 4.Developing resolution enhancement algorithms for satellite images.

26. 27 Questions?