Here today. Gone Tomorrow Aaron McClennon-Sowchuk, Michail Greshischev

Objectives  remove an object from a set of images by using information (pixels) from other images in the set.  The images must be of the same scene but can vary in time of taken and/or perspective of scene. The allowed variance in time means objects may change location from one image to the next. Applications: stock photography, video surveillance, etc.

Steps 1. Read Images 2. Project images in same perspective 3. Align the images 4. Identify differences 5. Infill objects

Reading Images  How are images represented? –Matrices (M x N x P) –M is the width of the image –N is the height of the image –P is 1 or 3 depend on quality of image 1: binary (strictly or white) or gray-scale images 3: coloured images (3 components of colour: R,G,B)  What tools are capable of processing images? –Many to choose from but MatLab is ideal for matrices. –Hence the name Mat(rix) Lab(oratory)

Identifying differences  Possible Methods: 1. Direct subtraction 2. Structural Similarity Index (SSIM) 3. Complex Waveform SSIM

Identifying differences 1. Direct subtraction –Too good to be true!(way too much noise)

Identifying differences 2. Structural Similarity Index (SSIM) –Number 0-1 indicating how “similar” two pixels are. –1 indicates perfect match, 0 indicates no similarities at all –Number calculated based on: – Luminance, function of the mean intensity for gray-scale image – Contrast, function of std.dev of intensity for gray-scale image

Identifying differences  Once again, way too much noise.  SSIM map: 0  black pixel1  white pixel

–Concerns: –Identify regions to copy Calculate a bounding box (smallest area surrounding entire blob) –How to distinguish noise from actual objects? Area - those blobs with area below threshold are ignored location - those blobs along an edge of image are ignored. –Copying method Direct – images from same perspectives Manipulated pixels – images from different perspectives. Infilling the objects

 Original bounding box results: Matlab returns Left position Top position Width and Height of each box

Infilling the objects  Result with small blobs and blobs along edges ignored:  Left: 119  Top: 52  Width: 122  Height: 264

Infilling the objects  Once regions identified, how can pixels be copied? –Same perspective – direct copy is possible.

Infilling the objects  Result of direct copying

Infilling the objects  Different perspectives –Goal: remove black trophy from left image

Infilling the objects  Direct copying produces horrendous results! Rectified image Result

Work to come...  Copying techniques –Need better method for infilling objects between images in different perspectives. Perhaps use same alignment matrix.  Anti-Aliasing –Method to smooth the edges around pixels copied from one image to another – example looks alright but could improve other test cases  User friendly interface –Current state: a dozen different MatLab scripts. –In the perfect world, we’d have a nice interface to let user load images and clearly displa

Conclusions

References  Z. Wang and A. C. Bovik, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Processing, vol. 13, pp. 600 – 612, Apr html html