Performance Testing “ Guide to Biometrics” - chapter 7 “ An Introduction to Evaluating Biometric Systems” by Phillips et al., IEEE Computer, February 2000,

Slides:



Advertisements
Similar presentations
Dr. Marc Valliant, VP & CTO
Advertisements

ECE 5367 – Presentation Prepared by: Adnan Khan Pulin Patel
Chapter 9 Creating and Maintaining Database Presented by Zhiming Liu Instructor: Dr. Bebis.
Chapter 8 Selecting a Biometric Presentation by Phani Dogiparthi.
1 Chapter 15 System Errors Revisited Ali Erol 10/19/2005.
Creating and Maintaining Databases Dr. Pushkin Kachroo.
Fingerprint Minutiae Matching Algorithm using Distance Histogram of Neighborhood Presented By: Neeraj Sharma M.S. student, Dongseo University, Pusan South.
C. L. Wilson Manager, Image Group Biometrics Overview of the PATRIOT Act.
IIIT Hyderabad Pose Invariant Palmprint Recognition Chhaya Methani and Anoop Namboodiri Centre for Visual Information Technology IIIT, Hyderabad, INDIA.
Confidence Measures for Speech Recognition Reza Sadraei.
66: Priyanka J. Sawant 67: Ayesha A. Upadhyay 75: Sumeet Sukthankar.
Section – Biometrics 1. Biometrics Biometric refers to any measure used to uniquely identify a person based on biological or physiological traits.
Cascaded Filtering For Biometric Identification Using Random Projection Atif Iqbal.
Biometrics & Security Tutorial 7. 1 (a) Please compare two different kinds of biometrics technologies: Retina and Iris. (P8:2-3)
Information Fusion in Multibiometric Systems
Keystroke Biometric : ROC Experiments Team Abhishek Kanchan Priyanka Ranadive Sagar Desai Pooja Malhotra Ning Wang.
Indexing and Binning Large Databases
FIT3105 Biometric based authentication and identity management
Introduction to Biometrics Dr. Pushkin Kachroo. New Field Face recognition from computer vision Speaker recognition from signal processing Finger prints.
GUIDE TO BIOMETRICS CHAPTER I & II September 7 th 2005 Presentation by Tamer Uz.
PALM VEIN TECHNOLOGY.
Chapter 11 Integration Information Instructor: Prof. G. Bebis Represented by Reza Fall 2005.
Chapter 5 Basic System Errors (Alireza Tavakkoli).
Comparison and Combination of Ear and Face Images in Appearance-Based Biometrics IEEE Trans on PAMI, VOL. 25, NO.9, 2003 Kyong Chang, Kevin W. Bowyer,
Robert S. Zack, Charles C. Tappert, and Sung-Hyuk Cha Pace University, New York Performance of a Long-Text-Input Keystroke Biometric Authentication System.
Biometrics and Authentication Shivani Kirubanandan.
05/06/2005CSIS © M. Gibbons On Evaluating Open Biometric Identification Systems Spring 2005 Michael Gibbons School of Computer Science & Information Systems.
Identification System Errors Guide to Biometrics – Chapter 6 Handbook of Fingerprint Recognition Presented By: Chris Miles.
Biometrics Kyle O'Meara April 14, Contents Introduction Specific Types of Biometrics Examples Personal Experience Questions.
A Brief Survey on Face Recognition Systems Amir Omidvarnia March 2007.
Authors: Anastasis Kounoudes, Anixi Antonakoudi, Vasilis Kekatos
Biometric Authentication Presenter: Yaoyu, Zhang Presenter: Yaoyu, Zhang.
1J. M. Kizza - Ethical And Social Issues Module 16: Biometrics Introduction and Definitions Introduction and Definitions The Biometrics Authentication.
Module 14: Biometrics Introduction and Definitions The Biometrics Authentication Process Biometric System Components The Future of Biometrics J. M. Kizza.
Uba Anydiewu, Shane Bilinski, Luis Garcia, Lauren Ragland, Debracca Thornton, Joe Tubesing, Kevin Chan, Steve Elliott, and Ben Petry EXAMINING INTRA-VISIT.
Biometrics: Ear Recognition
Karthiknathan Srinivasan Sanchit Aggarwal
1 Fingerprint Classification sections Fingerprint matching using transformation parameter clustering R. Germain et al, IEEE And Fingerprint Identification.
+ Hand Biometrics Measurement and biometric technologies for identification and security applications.
CS 736 A methodology for Analyzing the Performance of Authentication Protocol by Laseinde Olaoluwa Peter Department of Computer Science West Virginia.
1 Biometrics and the Department of Defense February 17, 2003.
CPSC 601 Lecture Week 5 Hand Geometry. Outline: 1.Hand Geometry as Biometrics 2.Methods Used for Recognition 3.Illustrations and Examples 4.Some Useful.
Introduction to Biometrics Charles Tappert Seidenberg School of CSIS, Pace University.
User Authentication Using Keystroke Dynamics Jeff Hieb & Kunal Pharas ECE 614 Spring 2005 University of Louisville.
BIOMETRICS By: Lucas Clay and Tim Myers. WHAT IS IT?  Biometrics are a method of uniquely identifying a person based on physical or behavioral traits.
Biometrics Stephen Schmidt Brian Miller Devin Reid.
Introduction to Biometrics Dr. Bhavani Thuraisingham The University of Texas at Dallas Lecture #23 Biometrics Standards - II November 14, 2005.
Lecture 7 Page 1 CS 236, Spring 2008 Challenge/Response Authentication Authentication by what questions you can answer correctly –Again, by what you know.
Biometrics Authentication Technology
Designing multiple biometric systems: Measure of ensemble effectiveness Allen Tang NTUIM.
By Hafez Barghouthi. Definition ”Biometric Technologies” are automated methods of verifying or recognizing the identity of a living person based on a.
Introduction to Biometrics Dr. Bhavani Thuraisingham The University of Texas at Dallas Lecture #5 Issues on Designing Biometric Systems September 7, 2005.
Biometric for Network Security. Finger Biometrics.
1 Figure 2-8: Access Cards Magnetic Stripe Cards Smart Cards  Have a microprocessor and RAM  More sophisticated than mag stripe cards  Release only.
GENDER AND AGE RECOGNITION FOR VIDEO ANALYTICS SOLUTION PRESENTED BY: SUBHASH REDDY JOLAPURAM.
Biometrics Chuck Cook Matthew Etten Jeremy Vaughn.
INTRODUCTION TO BIOMATRICS ACCESS CONTROL SYSTEM Prepared by: Jagruti Shrimali Guided by : Prof. Chirag Patel.
L. F. Coppenrath & Associates PASSWORD BIOPASSWORD ® Biometric Keystroke Dynamics Technology Overview.
An Introduction to Biometrics
Signature Recognition Using Neural Networks and Rule Based Decision Systems CSC 8810 Computational Intelligence Instructor Dr. Yanqing Zhang Presented.
BIOMETRICS VOICE RECOGNITION. Meaning Bios : LifeMetron : Measure Bios : LifeMetron : Measure Biometrics are used to identify the input sample when compared.
Challenge/Response Authentication
Lecture 7 Constructing hypotheses
Multimodal Biometric Security
FACE RECOGNITION TECHNOLOGY
FACE DETECTION USING ARTIFICIAL INTELLIGENCE
Biometrics.
Biometrics Reg: AMP/HNDIT/F/F/E/2013/067.
Biometrics.
Biometric technology.
Presentation transcript:

Performance Testing “ Guide to Biometrics” - chapter 7 “ An Introduction to Evaluating Biometric Systems” by Phillips et al., IEEE Computer, February 2000, pp Presented By: Xavier Palathingal September 21 st, 2005

Background Two biometric capabilities( matching and ranking) and biometric system errors Chapter 5 – 1:1 Biometric Matching Chapter 6 – 1:m Biometric Searching Relate “error quotes” to error definitions Look at accuracy numbers and reconstruct and interpret them

Overview Measuring performance Technology Evaluations Scenario Evaluations Operational Evaluations Comparison of the methods Limits to Evaluations

Overview (cont…) Implications of error rates Biometric Authentication - “Why does it reject me?” Biometric Screening – “Why does it point to me?” Face, Finger and Voice Iris and Hand geometry Signature Summary of verification accuracies

Overview (cont…) Identification System Testing Biometric data and ROC and CMC Biometric search engines 1:m search engine testing Face Recognition and Verification Test 2000 [ FVRT 2000 ] FVRT 2002 Face, Finger and Voice

Measuring Performance Evaluation protocols Why measure performance ? Determines how you test the system, select the data and measure performance Evaluation shouldn’t be too hard or too easy Is just right when it spreads performance over a range that lets to distinguish

Measuring Performance 1 -Technology Evaluations On laboratory or prototype algorithms “testing on databases” Move from general to specific “training” data A Sequestered “test” data Q Two phases Training phase Competitive testing phase

Phase 1 of Technology Evaluation- Training phase The algorithm is trained using “training” data A = (A 1 U A 2 U …) Then tested on newly made available sequestered “test” data Q

Phase 2 of Technology Evaluation - Competitive testing phase Using database D for each matcher Z, a set of match (genuine) scores X={X 1,X 2,…,X N } and a set of non-match scores Y={Y 1,Y 2,….,Y N } are generated. FMR and FNMR [FAR and FRR] are calculated as a function of threshold T

Measuring Performance 2 - Scenario Evaluations Tests complete biometric systems under conditions that model real world applications Combination of sensors and algorithms “office environment”, “user tests”

Measuring Performance 3 - Operational Evaluations Similar to scenario evaluations Scenario test – class of applications Operational test – specific algorithm for a specific application Performed at the actual site Using actual subjects/areas Usually not very repeatable

Comparison of methods Academia tend to use databases i.e.; technology evaluations acquisition procedures user population is closed in scenario evaluations Not “double blind” – technology and scenario

Limits to evaluation Biometric authentication should be mandatory to the whole user population User population should be fairly represented Subjects should be unaware of the matching decision Only realistic form of testing is operational evaluation One cannot measure the true FAR or true FNR – nobody except the actual subject knows Attempt to measure these “hidden” system parameters will be by trying to defeat the biometric system

Implication of error rates Biometric Authentication “Why does it reject me?” Verification protocol – frequent flyer smartcard with biometric - fingerprint template on a smartcard - unique frequent flyer no. and smartcard - FRR = 3% (typical for finger) people per hr [Newark airport] in a 14 hr day.03 x 5000 x 14 = will have to handled through exception handling procedures

Implication of error rates Biometric Screening “Why does it point to me?” Screening protocol – passenger face images with government face image database - a system that checks a face against a negative database N of n=25 alleged terrorists - FPR = 0.1% people request access to a flight 25 x 300 = 7500 matches 7500 x.001 = 7 false positives

Implication of error rates Biometric Screening “Why does it point to me?” The no. of false positives, FPR(n) ≈ n x FPR(1) Matching a positive data set M of m subjects requires m matches against a database N of n terrorists m = 300 n = 25 # false positives for plane = m x FPR(n) = m x n x FPR(1)

Face, Finger and Voice Technology evaluations FARs are operating around 10%

Iris “normal office environment”, with 200 volunteers over a period of 3 months In identification mode, not in verification mode High FRR may be due to environmental error, reflection from glasses, user difficulty

Hand Geometry Group of 50 users. 200 volunteers over a 3 month period

Signature Does not have the characteristic of permanence Accept = genuine, reject = forgery Zero-effort forgery, Home-improved forgery, Over-the-shoulder forgery, Professional forgery

Signature (cont….) Improvement of two-try over one-try indicates poor habituation of the biometric on that particular device

Summary of verification accuracies Best error rates found in literature One main thing is the volume

Identification system testing: Biometric data and ROC,CMC Biometric capabilities like ranking and matching need to be developed by modeling biometric data and training using biometric data Two different biometric statistics – ROC and CMC ROC – measures the capabilities of a match engine s(B’,B) with some fixed t 0 or as a function of some operating threshold T CMC – measures the capabilities of a rank engine R((B 1,B 2 ),B’ l ) with ordered entries (B 1,B 2 ) € M and some unknown sample B’ l

Biometric search engines A hybrid approach - ranking followed scoring Input to the 1:m search engine - B’ l, the biometric sample Output - vector C K (B’ l ) =(ID (1),…ID (K) ) T The 1:m search engine with an enrollment database of M is defined as : C K = (B (1),B (2),….,B (K) ) T = (ID (1),ID (2),…,ID (K) ) T

Biometric search engines (cont…) A possible architecture: - A biometric rank engine which determine some reordering C m of vector M by repeatedly applying ranking - A biometric match engine determine using a scoring function s(B’ l,B (k) ) and decision threshold t 0 (B’ l ),a short candidate vector C K of the K top candidates

1:m search engine testing The big distinction of a 1:m search engine compared to a 1:1 matcher - prerequisite of an enrollment database M = (B 1,B 2,….B M ) T We select the first m samples as database samples [9] For other samples, denoted as {B’ l,l=m+1}= D\M, a rank ř(B’ l ) is estimated as follows: 1.Computes the sets of scores X l = {s(B’ l,B i ); i = 1,….,m} for l = m + 1

1:m search engine testing (cont..) 2. Sort these scores: X ~ l = (s(B’ l,B (1) ),s(B’ l,B (2) ),….s(B’ l,B (m) )) T such that s(B’ l,B (k) ) > s(B’ l,B (k+1) ), 1 ≤ k < m 3. If (B’ l,B (k) ) is the mated pair, i.e., if B i = B (k) matches B’ l, ř(B’ l ) = k

Face Recognition and Verification Test 2000 First attempt to characterize performance measures 5 participating vendors had to compute an all-against-all match of a database of 13,872 face images Some results: 1.Compression does not affect performance adversely 2.Pose changes up to 25 degrees was handled by algorithms, beyond 40 the performance degrades sharply 3.Images taken 12 or more months apart are difficult to recognize 4.Distance between camera and person matters a lot 5.Identification is more sensitive to expression changes than verification is

FRVT 2002 An increase in database size Difference in results in plain verification tasks – K =sorted list size, m =gallery size

Thank you ! Especially to: Dr.Bebis for suggesting the additional paper Reza and Chang for help with the scanner