Data-driven Methods: Faces 15-463: Computational Photography Alexei Efros, CMU, Fall 2012 Portrait of Piotr Gibas © Joaquin Rosales Gomez.

Slides:



Advertisements
Similar presentations
Face Recognition Sumitha Balasuriya.
Advertisements

EigenFaces and EigenPatches Useful model of variation in a region –Region must be fixed shape (eg rectangle) Developed for face recognition Generalised.
Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs.
Figure-centric averages Antonio Torralba & Aude Oliva (2002) Averages: Hundreds of images containing a person are averaged to reveal regularities in the.
Feature-Based Image Metamorphosis Thaddeus Beier Shawn Neely SIGGRAPH 1992.
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm Lecture #20.
Dimensionality Reduction Chapter 3 (Duda et al.) – Section 3.8
Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004.
© 2003 by Davi GeigerComputer Vision September 2003 L1.1 Face Recognition Recognized Person Face Recognition.
Data-driven Methods: Faces : Computational Photography Alexei Efros, CMU, Fall 2007 Portrait of Piotr Gibas © Joaquin Rosales Gomez.
Face detection and recognition Many slides adapted from K. Grauman and D. Lowe.
Face Recognition using PCA (Eigenfaces) and LDA (Fisherfaces)
Face Recognition Based on 3D Shape Estimation
Face Recognition Jeremy Wyatt.
Face Recognition Using Eigenfaces
Principal Component Analysis Barnabás Póczos University of Alberta Nov 24, 2009 B: Chapter 12 HRF: Chapter 14.5.
Active Appearance Models Computer examples A. Torralba T. F. Cootes, C.J. Taylor, G. J. Edwards M. B. Stegmann.
Face Recognition from Face Motion Manifolds using Robust Kernel RAD Ognjen Arandjelović Roberto Cipolla Funded by Toshiba Corp. and Trinity College, Cambridge.
EECE 279: Real-Time Systems Design Vanderbilt University Ames Brown & Jason Cherry MATCH! Real-Time Facial Recognition.
Computer Vision I Instructor: Prof. Ko Nishino. Today How do we recognize objects in images?
Face Collections : Rendering and Image Processing Alexei Efros.
Faces: Analysis and Synthesis Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski.
Face Recognition: An Introduction
Learning the Appearance of Faces: A Unifying Approach for the Analysis and Synthesis of Images. Thomas Vetter Germany University of Freiburg
Understanding Faces Computational Photography
Face Detection and Recognition
Facial Recognition CSE 391 Kris Lord.
CS 485/685 Computer Vision Face Recognition Using Principal Components Analysis (PCA) M. Turk, A. Pentland, "Eigenfaces for Recognition", Journal of Cognitive.
Dimensionality Reduction: Principal Components Analysis Optional Reading: Smith, A Tutorial on Principal Components Analysis (linked to class webpage)
Recognition Part II Ali Farhadi CSE 455.
Face Recognition and Feature Subspaces
Last tuesday, you talked about active shape models Data set of 1,500 hand-labeled faces 20 facial features (eyes, eye brows, nose, mouth, chin) Train 40.
Face Recognition and Feature Subspaces
Face Detection and Recognition Computational Photography Derek Hoiem, University of Illinois Lecture by Kevin Karsch 12/3/13 Chuck Close, self portrait.
Internet-scale Imagery for Graphics and Vision James Hays cs195g Computational Photography Brown University, Spring 2010.
CSE554AlignmentSlide 1 CSE 554 Lecture 5: Alignment Fall 2011.
Digital Face Replacement in Photographs CSC2530F Project Presentation By: Shahzad Malik January 28, 2003.
1 Recognition by Appearance Appearance-based recognition is a competing paradigm to features and alignment. No features are extracted! Images are represented.
Face Recognition: An Introduction
CSE 185 Introduction to Computer Vision Face Recognition.
CSSE463: Image Recognition Day 27 This week This week Today: Applications of PCA Today: Applications of PCA Sunday night: project plans and prelim work.
Understanding Faces 12/1/15 Some slides from Amin Sadeghi, Lana Lazebnik, Silvio Savarese, Fei-Fei Li Chuck Close, self portrait Lucas by Chuck Close Detection,
Multimodal Interaction Dr. Mike Spann
CS 691B Computational Photography
CS 691B Computational Photography Instructor: Gianfranco Doretto Data Driven Methods: Faces.
Point Distribution Models Active Appearance Models Compilation based on: Dhruv Batra ECE CMU Tim Cootes Machester.
Active Appearance Models Dhruv Batra ECE CMU. Active Appearance Models 1.T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active Appearance Models", in Proc.
Face Recognition and Feature Subspaces Devi Parikh Virginia Tech 11/05/15 Slides borrowed from Derek Hoiem, who borrowed some slides from Lana Lazebnik,
Obama and Biden, McCain and Palin Face Recognition Using Eigenfaces Justin Li.
Face detection and recognition Many slides adapted from K. Grauman and D. Lowe.
CSSE463: Image Recognition Day 25 This week This week Today: Applications of PCA Today: Applications of PCA Sunday night: project plans and prelim work.
Multivariate statistical methods. Multivariate methods multivariate dataset – group of n objects, m variables (as a rule n>m, if possible). confirmation.
Face Detection & Recognition
CSE 554 Lecture 8: Alignment
CSSE463: Image Recognition Day 27
CSSE463: Image Recognition Day 26
University of Ioannina
Data-driven Methods: Faces
Lecture 8:Eigenfaces and Shared Features
Face Recognition and Feature Subspaces
Recognition: Face Recognition
Data-driven Methods: Faces
Outline Peter N. Belhumeur, Joao P. Hespanha, and David J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,”
Object Modeling with Layers
Computational Photography
PCA is “an orthogonal linear transformation that transfers the data to a new coordinate system such that the greatest variance by any projection of the.
Eigenfaces for recognition (Turk & Pentland)
CSSE463: Image Recognition Day 25
CSSE463: Image Recognition Day 25
Presentation transcript:

Data-driven Methods: Faces : Computational Photography Alexei Efros, CMU, Fall 2012 Portrait of Piotr Gibas © Joaquin Rosales Gomez

The Power of Averaging

8-hour exposure © Atta Kim

Fun with long exposures Photos by Fredo Durand

More fun with exposures

Figure-centric averages Antonio Torralba & Aude Oliva (2002) Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images.

More by Jason Salavon More at:

“100 Special Moments” by Jason Salavon Why blurry?

Computing Means Two Requirements: Alignment of objects Objects must span a subspace Useful concepts: Subpopulation means Deviations from the mean

Images as Vectors = m n n*m

Vector Mean: Importance of Alignment = m n n*m = ½ + ½ = mean image

How to align faces?

Shape Vector = 43 Provides alignment!

Average Face 1. Warp to mean shape 2. Average pixels

Objects must span a subspace (1,0) (0,1) (.5,.5)

Example Does not span a subspace mean

Subpopulation means Examples: Happy faces Young faces Asian faces Etc. Sunny days Rainy days Etc. Average male Average female

Deviations from the mean - = Image X Mean X  X = X - X

Deviations from the mean += = X  X = X - X

Manipulating Facial Appearance through Shape and Color Duncan A. Rowland and David I. Perrett St Andrews University IEEE CG&A, September 1995

Face Modeling Compute average faces (color and shape) Compute deviations between male and female (vector and color differences)

Changing gender Deform shape and/or color of an input face in the direction of “more female” original shape colorboth

Enhancing gender more same original androgynous more opposite

Changing age Face becomes “rounder” and “more textured” and “grayer” original shape colorboth

Back to the Subspace

Linear Subspace: convex combinations Any new image X can be obtained as weighted sum of stored “basis” images. Our old friend, change of basis! What are the new coordinates of X?

The Morphable Face Model The actual structure of a face is captured in the shape vector S = (x 1, y 1, x 2, …, y n ) T, containing the (x, y) coordinates of the n vertices of a face, and the appearance (texture) vector T = (R 1, G 1, B 1, R 2, …, G n, B n ) T, containing the color values of the mean-warped face image. Shape S Appearance T

The Morphable face model Again, assuming that we have m such vector pairs in full correspondence, we can form new shapes S model and new appearances T model as: If number of basis faces m is large enough to span the face subspace then: Any new face can be represented as a pair of vectors (  1,  2  m ) T and (  1,  2  m ) T !

Issues: 1.How many basis images is enough? 2.Which ones should they be? 3.What if some variations are more important than others? E.g. corners of mouth carry much more information than haircut Need a way to obtain basis images automatically, in order of importance! But what’s important?

Principal Component Analysis Given a point set, in an M -dim space, PCA finds a basis such that coefficients of the point set in that basis are uncorrelated first r < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r ) x1x1 x0x0 x1x1 x0x0 1 st principal component 2 nd principal component

PCA via Singular Value Decomposition [u,s,v] = svd(A);

Principal Component Analysis Choosing subspace dimension r : look at decay of the eigenvalues as a function of r Larger r means lower expected error in the subspace data approximation rM1 eigenvalues

EigenFaces First popular use of PCA on images was for modeling and recognition of faces [Kirby and Sirovich, 1990, Turk and Pentland, 1991]  Collect a face ensemble  Normalize for contrast, scale, & orientation.  Remove backgrounds  Apply PCA & choose the first N eigen-images that account for most of the variance of the data. mean face lighting variation

First 3 Shape Basis Mean appearance

Using 3D Geometry: Blinz & Vetter, 1999 show SIGGRAPH video

Walking in the Face-graph! Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg, Steven M. Seitz. "Exploring Photobios." ACM Transactions on Graphics 30(4) (SIGGRAPH), Aug

Photobio

Challenges Non-rigid (facial expressions, age…) Occlusions (hair, glasses …) Arbitrary lighting, pose Different cameras, exposure, focus … But: there are many photos!

Image registration Face detection Bourdev and Brandt ‘05 Fiducial points detection Everingham et al. ‘06 2D registration Template 3D model Estimate 3D pose Kemelmacher, Shechtman, Garg, Seitz, Exploring Photobios, SIGGRAPH’11

Image registration Face detection Bourdev and Brandt ‘05 Fiducial points detection Everingham et al. ‘06 3D registration Template 3D model Estimate 3D pose Kemelmacher, Shechtman, Garg, Seitz, Exploring Photobios, SIGGRAPH’11

3D transformed photos before after … …

Represent the photo collection as a graph 3D Head Pose similarity 3D Head Pose similarity Facial Expression similarity Time similarity Similarity between 2 photos

Represent the photo collection as a graph 3D Head Pose similarity 3D Head Pose similarity Facial Expression similarity Time similarity Similarity between 2 photos

Represent the photo collection as a graph 3D Head Pose similarity 3D Head Pose similarity Facial Expression similarity Time similarity Similarity between 2 photos