1. A Robust Super Resolution Method for Images of 3D Scenes Pablo L. Sala Department of Computer Science University of Toronto

2. Super Resolution “The process of obtaining, from a set of images that overlap, a new image that in the regions of overlapping has a higher resolution than that of each individual image.” This work: Introduces a method for robust super resolution Applies it to obtain higher resolution images of a 3D scene from a set of calibrated low-resolution images under the assumption that the scene can be approximated by planes lying in 3D space.

3. The Registration Problem The algorithm presented here starts from a discreet set of planes in 3D space and for each plane it back-projects all the input images onto that plane and performs super- resolution. Only the image pixels of 3D points of the scene that lie on the plane will be back- projected to the same points in the plane.

4. Problem Formulation Assumptions: The scene can be thought of as mostly lying on planes in 3D space. All viewpoints are located in a region separable by a plane from the scene, so to hold to a notion of scene depth. The Idea: A set of N calibrated images of the scene. A list of planes ordered by its depth. Images set is partitioned in disjoint subsets of relatively same size. Super-resolution attempted on each plane using the back-projection of the images of each subset on that plane. The obtained higher resolution images are compared. The regions of coincidence will be assumed to correspond to parts of the scene lying on the plane. The portions of each image that correspond to these regions will not be taken into account when applying super-resolution on the following planes.

5. Standard Super-Resolution Super-resolution can be formulated as an optimization problem: X is the unknown higher resolution image x j are the low resolution images F j = D j H j W j are the imaging formation matrices, with D j a decimation, H j a blurring, and W j a geometric warp matrix. Deriving E with respect to X and equaling it to 0 we get to:

6. Robust Super-Resolution Objective function: Error: Cauchy’s robust estimator:

7. Robust Super-Resolution Deriving E with respect to X: where W j is a diagonal matrix such that: Iteratively re-weighted least squares: Initial guess X 0 : the higher resolution image computed using the standard version of super-resolution.

8. The Algorithm Input: L a list of planes in 3D space; I j images; P j camera matrices Set O =  (set of the occluding 3D points) Randomly partition the images set into two or more disjoint subsets S 1,..., S R. For each plane  in L do: For each subset S i do: Use , P j and O to compute the image formation matrices F j X 0 = S-Resolution of images in S i Y i = Robust S-R of images in S i, using X 0 as the initial guess R = region R of  where all Y i coincide Assume R is the portion of the scene that lies on  O = O  R

9. Observations For the algorithm to be effective: The 3D planes uniformly distributed, close one to another, and covering all scene depth, so to detect all scene portions to be aware of occlusions. The scene should be highly textured for the similarity check to work well.

10. Experimental Results Synthetic scene:

11. 12 images were used as input 2 disjoint subsets Experimental Results

12. Standard vs Robust Super-Resolution: Experimental Results

13. Output images: Experimental Results

14. Results get worse as we advance in the 3D planes due to errors in defining precisely what portion of the scene lies in each plane Using more images and partitioning the images set in more than just two subsets will definitely give more accuracy to the similarity check. Using color images instead of just B&W. The super- resolution steps of the algorithm would be applied to each color channel separately and similarity would be assumed when the difference in reconstruction is close to zero for all three channels. Observations and Future Work