Digital Face Replacement in Photographs CSC2530F Project Presentation By: Shahzad Malik January 28, 2003

Face Replacement Motivation Currently done manually by graphic artists using photo editing software Currently done manually by graphic artists using photo editing software An automatic system has many potential uses: An automatic system has many potential uses: Hollywood special effects Hollywood special effects “Personalized” movies “Personalized” movies Framing someone… Framing someone…

Required Components Need the following subsystems: Need the following subsystems: Face detection (and tracking for videos) Face detection (and tracking for videos) Head pose estimator Head pose estimator Illumination extractor (*) Illumination extractor (*) Facial expression synthesis Facial expression synthesis Merging/replacement algorithm (*) Merging/replacement algorithm (*)

Light Estimation Assuming a Lambertian reflectance model: Assuming a Lambertian reflectance model: Any image can then be represented by: Any image can then be represented by:

Approximate Skin Tone Cannot assume 3 basis images for arbitrary photographs Cannot assume 3 basis images for arbitrary photographs Use an approximate image to generate basis Use an approximate image to generate basis

Fitting a Generic 3D Model Need geometry to create basis images Need geometry to create basis images Fit a generic 3D face mesh to images Fit a generic 3D face mesh to images “Lift” a texture using planar mapping “Lift” a texture using planar mapping

Generate Basis Images Set 3 linearly independent light positions Set 3 linearly independent light positions Relight skin tone model with each light Relight skin tone model with each light

Determining the Coefficients Compute a least squares solution to: Compute a least squares solution to: Solve separately for each RGB channel Solve separately for each RGB channel

Re-illuminating the Target Face Set intensities of the 3 light sources to the coefficient values Set intensities of the 3 light sources to the coefficient values Render the target face with these lights Render the target face with these lights

Flesh Pixel Detection Match non-mesh skin pixels to new skin tone Match non-mesh skin pixels to new skin tone Use a histogram-based skin classifier Use a histogram-based skin classifier

Histogram Matching Generate histograms for newly lit face Generate histograms for newly lit face Match the Gaussian distribution from original face to newly lit face Match the Gaussian distribution from original face to newly lit face For each flesh pixel in original image, choose a new color with a similar location on the Gaussian bell curve For each flesh pixel in original image, choose a new color with a similar location on the Gaussian bell curve

Weighted Color Blending Blend converted flesh pixels with face mesh pixels Blend converted flesh pixels with face mesh pixels

Results

Results (continued)

Summary Presented a face replacement system Presented a face replacement system Takes lighting and merging into account Takes lighting and merging into account Future research areas: Future research areas: Face detection and tracking (for videos) Face detection and tracking (for videos) Expression synthesis Expression synthesis More sophisticated reflectance model More sophisticated reflectance model Automatic and precise model-fitting Automatic and precise model-fitting