Matte-Based Restoration of Vintage Video 指導老師 : 張元翔 主講人員 : 鄭功運

outline Introduction..…………………………. 3 Prior Work……………………………..4 Our Approach To Video Restoration…..5 Result…………………………………..12

Introduction Film archives are worth preserving because of their historical and cultural values. Because of film deterioration over time, making it very fragile. Film deterioration and its attendant artifacts are in turn caused by aging and chemical decomposition, and improper storage and handling. Film artifacts can be categorized as - 1. film wear (scratches, film hair, blotches, dust, line jitter) 2. luminance-chrominance (color grading, color fading, and luminance flicker) 3. Image instability loss of resolution 4. noise contamination. Fig.1

Prior Work Some of the most common artifacts in vintage video are blotches and scratches. Most approaches assume the artifact areas have been totally corrupted; such areas are identified and replaced by pixels in neighboring frames. In addition, such approaches assume artifacts can be identified in the current frame using only the previous and next frames. Assuming the missing areas have been correctly identified,techniques have been proposed to fill them. Fig. 1. This source of information provides an important constraint on artifact removal that enables a restored image to closely match the actual scene. In our work, we make use of as much information that is available in the digitized video for restoration.

Our Approach To Video Restoration What complicates the digital restoration process is the generative models for different artifacts (e.g., lines versus blotches) are likely to be different. In our matting model, each pixel in the corrupted frames is assumed to be the linear interpolation of clean pixel and artifact colors. A : artifact colors α : the matte that indicates how much of the signal is in the observed data The darker the pixel (smaller α ), the more contaminated the observed pixel. Note that “*” denotes pixel-wise multiplication.

Given (typically 5 in our work) consecutive frames in video, our goal is to find an optimal linear combination of the true video and color artifact, together with a matte α in [0,1], so that (1) or, on a per pixel basis (2) with x being a 3-tuple indexing space and time. So, the true pixel color is given by (3) Before we describe our restoration inference framework, we first define, from (3) (4) as the hypothesized alpha-premultiplied true color. It can be computed given the hypotheses of and without getting unstable (due to division by zero). will become useful later in both the 3-D spatio-temporal CRF and its inference method

The dotted box (flow estimation) is optional if there is very little motion in the scene between frames. We apply affine global motion estimation at the earlier stages, and optionally apply local motion estimation for refinement.

Goal : recover the clean frames P, alpha maps α, artifact color maps A, and motion field D, given corrupted frames G. It can be generally formulated as – We extract A and α given D and G then treat our input video as a CRF, with each pixel as a node. Unfortunately, is generally intractable. So – we wish to maximize it. This has the advantage of recovering from fewer number of states.

U : the bias on alpha B : the smoothness potential. ω u 、 ω b : weights. (Note that the smoothness potentials depend on the observed data.) The bias term for α is - Smoothness potential -

E2 : encouraging the clean frames to be consistent spatially E3 : encouraging the clean frames to be consistent temporally to measure the spatial and temporal consistency among clean frames, we should be comparing the hypothesized true colors and by taking their difference: to avoid division by small (noisy) α ’s, we premultiply this term α y α x by to yield the difference term α x q y – α y q x. E4 : encourages α to be continuous, E5 : encourages the artifact color distribution A to be continuous : with ω 1 = 6.2 ω 2 = 1.5 ω 3 = 2.44 ω 4 = 0.01 ω 5 = 1.0

Given α and A, we can then attempt to recover P. However, it is unwise to compute directly Estimate the true color of a pixel using its immediate spatio-temporal neighborhood. The estimated restored color at x is –

Result

Advantage : 適用於任何的汙損回復，且將每個 frame 裡的所有資訊都用到以利於回復 Disadvantage : 非常耗時 – n : the number of pixels in 5 frames k : the total number of possible states (i.e., 22 or 33) for each pixel T : the number of iterations ~The End~