1. Automatic Face Recognition for Film Character Retrieval in Feature-Length Films
Ognjen Arandjelović Andrew Zisserman

2. “Groundhog Day” [Ramis, 1993]
The objective
Retrieve all shots in a video, e.g. a feature length film, containing a particular person
Visually defined search – on faces
Results shown on entire feature length movies including, Run Lola Run, Groundhog Day
For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie
You know the clock is very important.
“Groundhog Day” [Ramis, 1993]
Applications:
intelligent fast forward on characters
pull out all videos of “x” from 1000s of digital camera mpegs

3. The difficulty of face recognition
Image variations due to:
pose/scale
lighting
expression
partial occlusion

4. Previous work
There’s been significant progress in face recognition in the recent years:
Pose/illumination invariant recognition (e.g. The 3D Morphable Model – [Blanz et al., 2002])
Local feature-based approaches (e.g. Elastic Bunch Graph Matching – [Bolme, 2003], Sivic et al., 2005)
Appearance manifold-based methods and online appearance model building (e.g. see previous talk)
Etc.

5. System overview Five key steps: Feature localization Affine warping
Face outline detection
Refine registration
Robust distance

6. Facial feature detection
Train support vector machines to detect the eyes and the mouth (similar to “Names and Faces in the News” [Berg et al., 2004])
Independent Gaussian priors on feature locations
Example training data:
Learn invariance to:
pose
expression

7. Detected eyes and mouths
Successful detections in spite of large pose and expression variation

8. Warped faces using detected features
Original detected faces
Faces after affine warping

9. Background removal Key features and ideas: we do not use colour
only gradient information is used
faces are smooth with limited shape variability
model boundary traversal as a Markov chain
Significant clutter in images of detected faces

10. Background removal Radial mesh Solved using dynamic programming
Image intensity – threshold gradient to find interest points

11. Background removal – examples
Registered
Segmented

12. Registration refinement
faces already affine registered using 3 facial features
feature localization errors amount to a significant registration error
refinement using appearance – normalized cross-correlation of salient regions
Salient regions
Face 1
Face 2
Face 1 registered to 2

13. Occlusion detection Key points: Two faces being compared
occlusion detected when a pair of images is compared
from a training corpus learn the intra/intra-personal variance of each location/pixel
occlusion = pixels with low intra/inter-personal probability
contribution of occlusions to distance limited by Blake- Zisserman function
Two faces being compared
High occlusion probability
Hand
Grimace

14. Evaluation - querying The protocol: faces are automatically detected
query consists of one or more faces of the reference actor
and, optionally
images of non-reference actors

15. Evaluation - distances
Other
Three matching methods:
K-min distance
Linear subspace (reference only)
Nearest linear subspace (reference and other)
Correct person
Query

16. Evaluation - performance
Google-like retrieval, faces are ordered in decreasing similarity
Performance measure:
operates on sequences of recalled images
rank-ordering score S
in the range [0,1]
= 1 indicates all N true positives are recalled first
= 0.5 indicates a random ordering

17. Results - data Method evaluated on several films: Typical input data
Groundhog Day
Pretty Woman
Run, Lola Run
Fawlty Towers
Typical input data

18. Results – rank ordering score
Rank ordering score for 35 retrievals of Basil and Sybil
Basil
Sybil

19. Results – example recalls
Fawlty Towers
(John Cleese)
Pretty Woman
(Julia Roberts)

20. Results – example recalls
Groundhog day
(Andie MacDowel)
Groundhog day
(Bill Murray)

21. Conclusions Future work:
Use of sequence information for disambiguation in recognition (see “Person spotting: video shot retrieval for face sets” [Sivic et al., CIVR 2005])
Use of photometric models for improved illumination normalization