1. Page 1 The CSU Face Identification Evaluation System, ICVS 2003 Talk The CSU Face Identification Evaluation System: Its Purpose, Features and Structure David S. Bolme, J. Ross Beveridge, Marcio Teixeira and Bruce A. Draper Computer Science, Colorado State University 3rd International Conference on Computer Vision Systems - ICVS 2003

2. Page 2 The CSU Face Identification Evaluation System, ICVS 2003 Talk Goals of the CSU Face Recognition Evaluation Work Baseline/control Face Recognition algorithms. –Four algorithms selected from FERET 96/97 study. PCA, Eigenfaces (Turk and Pentland, MIT) PCA+LDA, (Zhao et. al., Maryland) Bayesian Image diff. Classifier, (Moghaddam et. al., MIT) Elastic Bunch Graph(Okada, et. al., USC) –Reference implementations in ANSI C. CSU Face Identification Evaluation System Statistical methodology for studying algorithms. –Parametric and Nonparametric methods –Standardized protocols and associated scripts. Determine critical factors that influence performance.

3. Page 3 The CSU Face Identification Evaluation System, ICVS 2003 Talk Obtaining the CSU Face Identification Evaluation System The Evaluation of Face Recognition Algorithms Website. First release of code on March 1, 2001 Current code release, –Version 4.0, –October 31, 2002 Over 1,500 downloads of Version 4.0 through March 2003 Users Guide is included and also available separately. http://www.cs.colostate.edu/evalfacerec/algorithms4.html

4. Page 4 The CSU Face Identification Evaluation System, ICVS 2003 Talk This ICVS 2003 Paper overlaps parts of the User’s Guide CSU Face Identification Evaluation System: Users Guide Installation Testing the system –Scripts,scrapshots System Overview –Image Formats –Distance Files Image Preprocessing Algorithms –PCA, –PCA+LDA, –BIC Analysis –Cumulative Match Curves –Error bars & distributions

5. Page 5 The CSU Face Identification Evaluation System, ICVS 2003 Talk System Overview Subspace Training Subspace Project Rank Curve Testing Permutation Testing Preprocessing Training Testing Analysis Standard Cumulative Match Curves Probability Distribution for Recognition Rate Normalization Bayesian Training Bayesian Project

6. Page 6 The CSU Face Identification Evaluation System, ICVS 2003 Talk Refinement of NIST preprocessing used in FERET. Image Preprocessing Integer to float conversion –Converts 256 gray levels to single-floats Geometric Normalization –Aligns human chosen eye coordinates Masking –Crop with elliptical mask leaving only face visible. Histogram Equalization –Histogram equalizes unmasked pixels: 256 levels. Pixel normalization –Shift and scale pixel values so mean pixel value is zero and standard deviation over all pixels is one.

7. Page 7 The CSU Face Identification Evaluation System, ICVS 2003 Talk The csuSubspace module: PCA and PCA+LDA … PCA+LDA space projection Distance Matrix Training images Eigenspace Combined space (PCA+LDA) Training Testing …

8. Page 8 The CSU Face Identification Evaluation System, ICVS 2003 Talk Bayesian Image difference Classifier: Take Difference of Images  Classify difference image as either: Intrapersonal from same subject Extrapersonal from different subjects Intrapersonal Example Extrapersonal Example - - = =

9. Page 9 The CSU Face Identification Evaluation System, ICVS 2003 Talk Bayesian Image difference Classifier: Training Uses csuSubspace Module csuMakeDiffs csuSubspaceTrain... Extrapersonal... Intrapersonal All Training Images Extrapersonal PCA Subspace Intrapersonal PCA Subspace

10. Page 10 The CSU Face Identification Evaluation System, ICVS 2003 Talk Bayesian Image difference Classifier: Testing uses csuBayesianProject Extrapersonal PCA Subspace Intrapersonal PCA Subspace CsuBayesianProject Probe & Gallery Images Distance Matrix

11. Page 11 The CSU Face Identification Evaluation System, ICVS 2003 Talk Evaluation Methodology and Tools Two Distinct Questions 1.Is an observed difference in performance significant? Monte Carlo Inference. Generalized Linear Models. 2.What covariates, and combinations of covariates, most influence performance? And how much? McNemar’s Test. Monte Carlo Inference. McNemar’sTest Tally when one algorithm succeeds and the other fails. Monte Carlo Inference Example: Sample Recognition Rate Probability Distribution created by perturbing probe gallery choice. Generalized Linear Model Example: Mixed Effects Logistic Regression with Repeated Measures on People. Power WeakStrong Complexity Simple Involved Covariates covers both features of algorithms and of people Version 4.0

12. Page 12 The CSU Face Identification Evaluation System, ICVS 2003 Talk Training, Probes, Galleries, What Varies? Training Gallery Probes F F F F FV F VF F VV F F F V V F VV V V V V Essentially FERET 1996/97 Micheals & Boult CVPR 2001 CSU PCA vs. PCA+LDA Analysis CSU PCA+LDA Configuration Analysis F V Fixed Throughout Study Varied, i.e. randomly sampled

13. Page 13 The CSU Face Identification Evaluation System, ICVS 2003 Talk Producing Cumulative Match Curves

14. Page 14 The CSU Face Identification Evaluation System, ICVS 2003 Talk Producing Sample Distributions TrainingTesting - Galleries and Probes Day 1Day 2 1 Subject 2 3 160 1 Subject 2 3 200 123 Id.1234 114PG 67PG 53PG 145PG 6GP 154GP 71GP 98GP … 99GP Balanced Sampling Compare PCA and PCA+LDA. Distance Measures: L1, L2, Mah. Angle (PCA), Soft L2 (PCA+LDA). Methodology: Monte Carlo Sampling of Probe/Gallery. CVPR 2001 citation.

15. Page 15 The CSU Face Identification Evaluation System, ICVS 2003 Talk PCA vs. PCA+LDA Confidence Intervals Sample Probability Distribution for PCA at rank 1 using Mahalanobis Distance. 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.500.520.540.560.580.590.610.630.650.670.690.710.730.75 Probability

16. Page 16 The CSU Face Identification Evaluation System, ICVS 2003 Talk Tabular Output from csuPermute

17. Page 17 The CSU Face Identification Evaluation System, ICVS 2003 Talk PCA vs. PCA+LDA Comparing Distance Measures Distance Measure Matters PCA favors Mahalanobis Angle PCA+LDA, Soft and Angle Similar Cumulative Match with Error Bars Distance choice more important than subspace.

18. Page 18 The CSU Face Identification Evaluation System, ICVS 2003 Talk Current Research FERET Subject Covariates Covariates for 2,974 Images, 1,209 Subjects

19. Page 19 The CSU Face Identification Evaluation System, ICVS 2003 Talk FERET Covariates Results (Preliminary!) Glasses Off Age Young Eyes Open Expression Neutral Race White No Facial Hair No Makeup Mouth Closed No Bangs Skin Clear Male Glasses Off/On Eyes Open/Closed Expression Changes Facial Hair Changes Always Makeup Makeup Changes Mouth Always Open Mouth Changes Bangs Change Skin Not Clear Glasses Always On Age Old Eyes Always Closed Always Non-neutral Race Asian Race African-Amer. Race Other Always Facial Hair Always Bangs Female -50%-40%-30%-20%-10%0% 10%20%30%40%50% Change in Similarity Measure Harder to Recognize Easier to Recognize

20. Page 20 The CSU Face Identification Evaluation System, ICVS 2003 Talk Conclusion Release 4.0 Contains –Three algorithms: PCA, PCA+LDA, BIC. –Cumulative match curve and probe gallery permutation tools. –Scripts for common experiments, including standard FERET. Supported platforms include –Code is ANSI C: Unix, Windows, … –Turn-key scripts and code tested on Linux, Solaris, Darwin. Over 1,500 downloads since October 31, 2002. Related papers on web site. Near Future - Release: 5.0 –Elastic Bunch Graph Matching (USC FERET). –Data Preparation for Generalized Linear Models. PCA+LDA Configuration and FERET Subject Covariate Study.

21. Page 21 The CSU Face Identification Evaluation System, ICVS 2003 Talk The End

22. Page 22 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for csuPreprocesNormalize

23. Page 23 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for SubspaceTrain

24. Page 24 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for csuSubspaceProject

25. Page 25 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for csuMakeDiffs First step in Bayesian Algorithm

26. Page 26 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for csuBayesianProject

27. Page 27 The CSU Face Identification Evaluation System, ICVS 2003 Talk Help for csuAnalysis Tools