Image Renaissance Using Discrete Optimization Cédric AllèneNikos Paragios ENPC – CERTIS ESIEE – A²SI ECP - MAS France

Image Renaissance Outline 1)Theory: positions and patches 2)Finding positions with good similarities 3)Determining the patch sizes 4)Results and discussions

Image Renaissance Theory: positions and patches

Image Renaissance 1 - Theory: positions and patches Main idea: inpainting as a jigsaw Inpainting:  Reconstructing a destroyed part of an image (specified by a mask)  The process should be undetectable Our method:  Considering the reconstruction as a jigsaw which pieces are patches taken from the “good” part of the image

Image Renaissance Two steps 1)Finding positions/offsets in the “good” image having strong similarities with the content of the image at the boundary of the area to rebuild 2)For each offset found, finding which pixels should be copied to reconstruct the destroyed area 1 - Theory: positions and patches

Image Renaissance Finding positions with good similarities

Image Renaissance Positions and offsets  Such a problem consists in finding, given an origin point at the boundary of the region to inpaint, a number of candidate positions in the image domain that have similar content with the one at the boundary of the area to rebuild  A few remarks:  Missing content is not necessarily in the neighborhood of the destroyed region…  Testing each offset in the image for each position at the boundary of the destroyed area would be too expensive in computation time…  Our solution: using a probabilistic method, the particle filters 2 – Finding positions with good similarities

Image Renaissance Particle filters theory  Considering a fixed number of samples, we have to repeat this steps:  Weight the samples with a Probability Density Function (pdf)  Generate a new set of samples (of equal number) from the existing ones with the biggest weights applying small perturbations (importance of perturbation decreases with the iterations to obtain a convergence) 2 – Finding positions with good similarities

Image Renaissance Particle filters in positions research  A particle has an origin point (at the boundary of the area to inpaint, which is invariant), an offset and a size (width and height)  The perturbation is a displacement of the offset and a modification of the size  The weight is evaluated by a Sum of Square Differences (SSD) between the sized neighborhood around the origin point and the offset point balanced with the size 2 – Finding positions with good similarities

Image Renaissance Determining the patch sizes

Image Renaissance How to find the optimal partition  For each pixel in the area to inpaint we have multiple offsets (and, as a consequence, multiple pixel values possible), so we have to find the best label partition to rebuild the image:  In that aim we use a combinatorial optimization: the α -expansion algorithm (using graph cuts) 3 – Determining the patch sizes

Image Renaissance Term 1 to minimize: constraints to data  : region of the image to be reconstructed  : region of the image to be reconstructed and its close neighborhood (which contains data)  : distance (similarity) between the label and the data of the image at position 3 – Determining the patch sizes

Image Renaissance Term 2 to minimize: constraints between patches  : region of the image to be reconstructed  : 4-neighborhood of  : distance (similarity) between the label L and the label at position  : Euclidian distance from to the boundary of the area to rebuild (to reconstruct firstly the borders of the region) 3 – Determining the patch sizes

Image Renaissance Term 3 to minimize: smoothing term  : region of the image to be reconstructed  : 4-neighborhood of  : Potts model: penalty for neighbors with different 3 – Determining the patch sizes

Image Renaissance Global energy function on partition to minimize ω  Minimized through successive α -expansion (using graph cuts) 3 – Determining the patch sizes

Image Renaissance Alpha-expansion principle  With a given partition, the algorithm expands the label α over all the other labels minimizing the energy function 3 – Determining the patch sizes "green"-expansion

Image Renaissance Alpha-expansion algorithm  For each offset selected (explaining the interest to reduce their number), we:  Construct a graph representing the energy on the area to inpaint  Execute the graph-cut algorithm on the graph  Modify the partition  This is repeated until stability (but in practice, one iteration on each offset gives a result near of the final result)  Each iteration will so enhance the result by minimizing the global energy Y. Boykov, O. Veksler and R. Zabih. Fast Approximate Energy Minimization via Graph Cuts. PAMI – Determining the patch sizes

Image Renaissance Results and discussions

Image Renaissance Results 4 – Results and discussion

Image Renaissance Results 4 – Results and discussion

Image Renaissance Conclusion  Graph-based combinatorial approach for inpainting  Advantages:  Efficient for textured images  Quite fast: in general a couple of minutes (but the final result is nearly reached after the first series of α -expansion)  Drawbacks:  Polynomial cost depending on the number of offsets selected (so particular attention must be paid on this step) 4 – Results and discussion

Image Renaissance Thank you for your attention! Any question?