A Sampled Texture Prior for Image Super-Resolution Lyndsey C. Pickup, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford Goal: Super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. Given some basic knowledge about the image subject, it should be possible to improve the super-resolution estimate by incorporating information available from similar images. Introduction The Sample-Based Prior There are K low-resolution images y (k), generated by y (k) = W (k) x +  G (k), where  G is a vector of i.i.d. Gaussians such that  G » N(0,  -1 ), and  G is the noise precision. If x has N pixels and each y (k) has M pixels, W (k) is an M£N matrix which encompasses the warping, blurring and decimation of x to produce y (k). We denote the registration and blurring parameters of y (k) by  (k). Low-resolution images (inputs) The goal is to reconstruct the high-resolution image to the right. It is assumed that the low-resolution images have been generated from this image through a process that blurs and subsamples the high-resolution image, and adds Gaussian noise. A few results Maximum Likelihood Solution The maximum likelihood images are very noisy because the problem can be ill-conditioned. where g is an approximation to the image gradient at each pixel in four different directions (horizontal, vertical, both diagonals), and  is the Huber function, defined Since this is an image of a fairly specific type, it should be possible to do better. Rather than using a generic Huber-style prior based on image statistics, samples from similar images can be used to build a more appropriate prior. An approach that has been shown to work well in texture synthesis is used to build the image prior. Rather than developing a parametric model, samples are taken from other images. Small sample image patches of edge length 5 to15 pixels are taken from a training image containing similar textures to the images being super-resolved. For each patch, neighbourhood information is stored along with a central pixel value. The patches are weighted by a 2D Gaussian so pixels near the centre carry more weight. To evaluate this texture-based prior Take each pixel x i in the super-resolution image x, and find its neighbourhood R(x i ). Find the closest matching neighbourhood in the patches sampled from the training image. Look up the central pixel value of that neighbourhood, L R (x i ). Assume x i is Gaussian distributed about L R (x i ) with precision  T. This gives us: Solving for the super-resolution image Start with an initial estimate of x, for instance the average image. We assume that the image registration has already been estimated accurately. Sample patches from an image in the same class as the super-resolution image being recovered. Store the neighbourhood and patch centre data for use with the texture-based prior. Use SCG to optimize the MAP version of the super-resolution model incorporating the new p(x): where  encompasses the  T and  G terms from the prior and data likelihood.  = 0.16  = 0.64  = 0.01  = 0.04 Original image Sample texture With a weak prior, the input data contains enough evidence for the prior to look like the ground truth, but as  increases, the prior term dominates. Maximum A Posteriori Solution Pixel neighbourhood Central pixel Using the wrong texture Text truth Text low-res Text training image Brick training image Brick truthBrick low-res Beads training image Beads truthBeads low-res A plot of root mean square (RMS) errors obtained from texture-based prior against those from the Huber-MAP images, where the error is calculated with respect to the ground truth images. In all cases, the texture- based prior’s image has a lower RMS error. For each image type, nine datasets were generated, using varying numbers of inputs and different noise levels. For each dataset, the best of several runs (while varying free model parameters) is shown. Note that training images do not overlap with image sections used as ground truth. The model Image x.Warp, with parameters  Blur by point-spread function. Decimate by zoom factor. Corrupt with additive Gaussian noise Noise (grey levels) Number of images Noise (grey levels) Number of images