Compressive Coded Aperture Video Reconstruction Roummel F. Marcia, Rebecca M. Willett European Signal Processing Conf. (EUSIPCO), August 2008

Outline Introduction Problem Formulation Compressive sensing Compressive coded aperture mask design Video setting Sparse Representation Algorithm Simulation

Introduction Many existing techniques for increasing video resolution are based on superresolution image reconstruction. These approaches, however, typically either require relatively large numbers of observed pixels or yield unsatisfactory edges and boundaries. This paper describes a novel technique for increasing the resolution of digital video using a combination of coded aperture sensing and wavelet-based reconstruction algorithms.

Introduction Coded aperture masks based on Toeplitz-structured matrices for compressive sensing can be used to reconstruct high resolution static images from low-resolution coded observations. There is a tradeoff between the sparsity of the vector being reconstructed and computation time. If each frame process independently, the problem size would be small and each iteration of a reconstruction method would require little computation time. If a block of sequential frames were processed simultaneously, problem size would increase and computation time also increase, but the solution could exploit inter-frame correlations, more sparse, and hence yield more accurate reconstructions

Problem Formulation Compressive sensing Compressive coded aperture mask design Video setting

Compressive Sensing High dimensional vectors can be recovered with high accuracy from a much smaller dimensional observation(y) when f has a “sparse” representation in some basis W.

Compressive Coded Aperture Mask Coded aperture imaging first desires to increase the light hitting a detector without sacrificing resolution. The basic idea is to create a mask pattern which introduces a more complicated point spread function, and exploit this pattern to reconstruct high-quality image estimates. A recent study addressed the accurate reconstruction of a high resolution static image which has a sparse representation in some basis from a single low resolution observation using compressive coded aperture imaging. “Compressive coded aperture superresolution image reconstruction”, ICASSP , 2008

Compressive Coded Aperture Mask Observation y is assumed： y = D(f * h) + n D is a downsampling operator and h is a point-spread function (PSF) can also be recognized as coded aperture patterns. Let F be the matrix corresponding to fast Fourier transform, and denote the Fourier transform of h by H (F(h) = H.), Let be a diagonal matrix whose diagonal components are the entries in H. Our goal is to design a mask pattern h such that the resulting image reconstruction is better than using no mask. This involves defining a pattern h such that the corresponding observation matrix R satisfies an RIP.

Compressive Coded Aperture Mask A method(by author) for randomly generating a mask h was developed so that the corresponding matrix product is block-circulant: Block-circulant matrices are known to be a compressed sensing matrix, and based upon recent theoretical work on Toeplitz-structured matrices for compressive sensing, the proposed masks are fast and memory-efficient to compute.

Video Setting The problem of accurately reconstructing a high resolution video from low resolution observations can be modeled mathematically by solving For judiciously chosen coded aperture masks p such that the corresponding observation matrix R satisfies the RIP, then solving a minimization problem will yield highly accurate reconstructions according to compressive sensing theory.

Sparse Representation Algorithm Our goal is to solve efficiently for a sequence of closely related video frame images Method A. For a scene that changes only slightly from frame to frame like adjacent frames, the reconstruction from a previous frame is often a good approximation to the following frame. So we use the solution at the frame as the initial value of for the optimization problem for the frame. Thus, few iterations will be needed for each optimization problem.

Sparse Representation Algorithm Method B and C method B and C are both improve method A by solving multiple frames simultaneously Solving two frames(method B2 and C2) The objective function is separable, two solutions and are not related.

Sparse Representation Algorithm Solve the couple of optimization instead： We compute and such that , then since , is very sparse compare to

Sparse Representation Algorithm Solving k>2 frames simultaneously in methods B and C Methods B and C are differ by and

Sparse Representation Algorithm Method B4 (4 frames)

Sparse Representation Algorithm Method C4 (4 frames) Sparsity may greater than B4

Simulations Optimization problems solving ：GPSR algorithm 50 256X256 gray-scale frames video depicting the movement of a vehicle on a street at the Duke University campus, but campus background remaining stationary. Low-resolution video image：each frame is downsampled by four in each dimension (resolution=64X64) Solving different number of frames : Methods A (1 frame), B2/C2 , B4 and C4 ,C8 and C12 Noise variance σ = 1.0 and 0.1

Simulations

Simulations Time limited for each frame：5 seconds and 20 seconds(only for σ=1.0) Observation matrix R and the basis matrix W multiplication take most of the computation time Method Iterations in 5 second/frame A 110 B2/C2 48 B4/C4 22 B8/C8 9 C12 4

Simulations

Conclusion This paper present compressive sensing methods for overcoming the pixel-limited resolution of digital video imaging systems. This method apply coded mask designs for each video frame and use sparse representation optimization techniques for signal recovery. If the desired accuracy is fixed, processing time required to achieve that accuracy is smaller when the block size (number of frames ) is larger, demonstrates the importance of exploiting inter-frame correlations, even when it increasing the size of the problem.