1. CUBSCenter for Unified Biometrics and Sensors, NY, USA 1 ER 2 : An Intuitive Similarity Measure for On-Line Signature Verification Hansheng Lei, Srinivas Palla, Venu Govindaraju CUBS, Center for Unified Biometrics and Sensors Univ. at Buffalo, the State Univ. of New York Amherst, NY USA 14260 {hlei, spalla2, govind}@cse.buffalo.edu

2. CUBSCenter for Unified Biometrics and Sensors, NY, USA 2 ER 2 : An Intuitive Similarity Measure for On-Line Signature Verification 1. Introduction- On-line signature verification 2. ER 2 : Intuitive Similarity Measure 3. Experimental Results 4. Demo – CUBS signature verification system 5. Conclusion 6. References

3. CUBSCenter for Unified Biometrics and Sensors, NY, USA 3 Introduction  Handwritten signatures are commonly used for financial transactions and documents.  Verification is usually done by visual inspection.  Unlike iris, retina, fingerprint, face, signature does not require any expensive hardware, thus it is already widely accepted by general public.  Two kinds of signatures: off-line and on-line. Fig.1 An on-line signature sensor. The X-Y coordinates and Pressure of signing are captured. With more sophisticated devices, Altitude and Azimuth are also recorded.

4. CUBSCenter for Unified Biometrics and Sensors, NY, USA 4 Introduction  Ideal Goals of On-line Verification 1. High accuracy (current accuracy is about 97% depending on test datasets) 2. Eliminating fraud. 3. Cheap implementation. 4. Substituting PIN or password. On-line signature verification is attracting increasing interests, academic and industrial.

5. CUBSCenter for Unified Biometrics and Sensors, NY, USA 5 Introduction  Challenges 1). Intra-class variation We are unaware of whether an individual ’ s signature is unique. The variation of a person ’ s signature can be large. 2). Forgery Easier to be forged than other biometric attributes such as fingerprint, iris, etc. 3). Very limited signatures for training Usually we can not expect more than 6 genuine signatures for training for each individual. This is unlike handwriting recognition. 4). Decide the consistent features There are possibly over 100 features for signature[2], such as Width, Height, Duration, Orientation, X positions, Y positions, Speed, Curvature, Pressure, so on. Which of them are reliable?

6. CUBSCenter for Unified Biometrics and Sensors, NY, USA 6 Introduction Basic Procedure for Signature Verification  Raw data Preprocessing Make signature invariant to scaling, translation & rotation.  Template generation from given signature The generated template include: 1)what kinds of feature are chosen, 2)the features,3) distance measures, 4) the threshold for decision.  Verification according to the template 1). Preprocess the raw data of the given signature. 2). Extract features and compare distances with the those in the template. 3). Make decision according to the threshold specified in the template.

7. CUBSCenter for Unified Biometrics and Sensors, NY, USA 7 Introduction - Raw data Preprocessing  Invariant to scaling and translation Suppose Sig=[X Y], both X and Y are sequences. To make it invariant to scaling and translation by mean- standard deviation normalization:  Invariant to rotation Method A. Represent sig=[X Y] in polar space. (x i, y i ) => (r i, θ i ). Method B. Determine the orientation of the mass of signature and rotate it.

8. CUBSCenter for Unified Biometrics and Sensors, NY, USA 8 Introduction - Raw data Preprocessing  Arc-length Normalization Given signature is considered as a 2D curve. It is believed that it is necessary to normalize its length and resample the points by equal arc-length.  Smoothing the curve Smoothing is to discard the noises. Basically two choices: 1). Gaussian filters. Convolute the curve with a Gaussian mask. 2). FFT transform. FFT makes energy concentrated on the first few coefficients. We can extract these coefficients and reversely FFT back to reconstruct the sequences.

9. CUBSCenter for Unified Biometrics and Sensors, NY, USA 9 Introduction - Raw data Preprocessing mean-std norm. Resampling Smoothing

10. CUBSCenter for Unified Biometrics and Sensors, NY, USA 10 Introduction -Template generation  Feature Extraction/Selection Because of limited training samples of signatures (say, 6) and no forgeries, features can not be extracted statistically. We think statistics-based methods are quite difficult.  Distance Measures Distance measures are associated with features. For scalar features, Euclidean norm is a proper measure; for sequential features, Dynamic Time Warping (DTW) is good measure.

11. CUBSCenter for Unified Biometrics and Sensors, NY, USA 11 Introduction -Template generation Features  Global features : #Width, Height, #Duration, #Orientation  Local features : #X-coordinates, #Y-coordinates, #Curvature  Dynamic features : #Velocity, #Acceleration, #Pressure, #Pressure changing  Other features : # Number of segments, #Critical points, etc.

12. CUBSCenter for Unified Biometrics and Sensors, NY, USA 12 Introduction -Template generation

13. CUBSCenter for Unified Biometrics and Sensors, NY, USA 13 Introduction -Template generation  Coordinate sequences X, Y, [X,Y ] are the most straightforward features. They are featureless features.  Speed sequences. Speed V, speed of X-coordinate V x and speed of Y- coordinate V y can be derived from sequence [X,Y ] directly by subtracting neighboring points. From the speed, acceleration V a can be further derived.

14. CUBSCenter for Unified Biometrics and Sensors, NY, USA 14 Introduction -Template generation  Pressure, Altitude, Azimuth Pressure is one of the most common dynamic information of on-line signature. Some devices can capture additional information, such as Azimuth (the clockwise rotation of cursor about the z-axis) and altitude (the angle upward toward the positive z-axis).

15. CUBSCenter for Unified Biometrics and Sensors, NY, USA 15 Introduction -Template generation  Center of Mass, Torque, Curvature-ellipse S 1 and S 2 The five features were defined by Vishvjit S. Nalwa [6]. Torque measures the area swept by the vector of pen position. S 1 and S 2 measure the curvature ellipse based on moments. The distance measure used here is cross- correlation (Pearson's r) weighted by the consistency of points.

16. CUBSCenter for Unified Biometrics and Sensors, NY, USA 16 Introduction -Template generation  Average, average positive speed on X- axis,average positive speed on Y-axis, total signing duration. Lee et al. [3] lists two sets of scalar features (over 100 features). These four features have the highest preference in the first set. The distance measure is Euclidean norm.

17. CUBSCenter for Unified Biometrics and Sensors, NY, USA 17 Introduction -Template generation  Cos(a), sin(a), Curvature a is the angle between the speed vector and the X-axis. The three features are proposed by Jain et al. [10]. It also proposes coordinate sequence differences.

18. CUBSCenter for Unified Biometrics and Sensors, NY, USA 18 Introduction -Template generation  Features (examples) A Signature sample X coordinatesY coordinates

19. CUBSCenter for Unified Biometrics and Sensors, NY, USA 19 Introduction -Template generation  Features (examples) Torques S 1 of Curvature ellipse A Signature sample

20. CUBSCenter for Unified Biometrics and Sensors, NY, USA 20 Introduction -Template generation  Feature comparison X-coordinates (genuine) X-coordinates (forgery) Genuine sig. Forgery sig. Only X-coordinates can not distinguish them!

21. CUBSCenter for Unified Biometrics and Sensors, NY, USA 21 Introduction -Template generation We have following experience: 1). One of the most reliable features is the shape of the signature. Shape is described by the combination of X, Y- coordinates [X,Y]. 2). The second reliable feature is the speed of writing. To represent shape and speed, each signature is a 3-D sequence: Sig i =[X i, Y i, V i ], where V is the sequence of speed magnitude. Then we use ER-Squared to match two signatures and return a Confidence of similarity (0%-100%). The details will be given later in section 2..

22. CUBSCenter for Unified Biometrics and Sensors, NY, USA 22 Introduction -Distance measures Most commonly-used measures  Euclidean norm  Weighted Correlation Where f(l), h(l) are functions of two signatures and w(l) is the consistency function.  Dynamic Time Warping (DTW) Elastic sequence matching. Very good for on-line signatures.

23. CUBSCenter for Unified Biometrics and Sensors, NY, USA 23 Introduction -Distance measures  DTW One-One alignmentDynamic alignment Both Euclidean norm and correlation assume one-one alignment. Easy but brittle! Elastic alignment is more robust for sequences, at the cost of computational resources.

24. CUBSCenter for Unified Biometrics and Sensors, NY, USA 24 Introduction -Distance measures  DTW Current cost Recursive cumulative cost The calculation of matrix D. The DTW warping path in the matrix D is the path which has minimum average cumulative cost. The unmarked area is the constraint imposed by |i-j|<w (w is the width of the allowed margin). Subject to optional constrain: |i-j|<w

25. CUBSCenter for Unified Biometrics and Sensors, NY, USA 25 Introduction - some remarks Remarks on some research directions in on-line signature verification  Segmentation? Signature is an art of drawing, not limited to some kind language. A Segmentation method by Perceptually Important Points was proposed by Jean-Jules Brault et al [7]. Many works have been done to apply segmentation to signature verification. Problems: 1)The consistency of segmentations? 2)If DTW is used as measure, Segmentation is of little necessity, because those Perceptually Important Points can be aligned accurately by DTW.

26. CUBSCenter for Unified Biometrics and Sensors, NY, USA 26 Introduction - some remarks  User-dependent distance threshold? Distance (Euclidean, DTW, etc.) for dissimilarity measure is not intuitive. In real applications, users tends to ask: how similar is the two signatures? Or, what is the confidence that this signature is genuine? It is intuitive to answer: their similarity confidence is 90%! (instead of saying their distance of dissimilarity is 5.8). It is hard to obtain a user-dependent threshold, because of limited genuine samples. Though it is a choice to use the genuine samples from other users as forgeries, it won ’ t help much on determining the threshold.

27. CUBSCenter for Unified Biometrics and Sensors, NY, USA 27 Introduction - some remarks  Statistics based methods? Again because we can not expect many signature samples, statistics based methods, such as Markov Model, is hard to achieve high performance. Artificially generate genuine signatures? Using random forgeries or use the signatures from other users? Possible ways.

28. CUBSCenter for Unified Biometrics and Sensors, NY, USA 28 ER 2 : An Intuitive Similarity Measure for On-Line Signature Verification 1. Introduction- On-line signature verification √ 2. ER 2 : Intuitive Similarity Measure 3. Experimental Results 4. Demo – CUBS signature verification system 5. Conclusion 6. References

29. CUBSCenter for Unified Biometrics and Sensors, NY, USA 29 ER 2 : Intuitive Similarity Measure Similarity measures must satisfy:  The similarity of intra-class is very high. (so that we can accept genuine signature)  The similarity of inter-class is very low. (so that we can reject forgery).  An intuitive score range, like 0 - 1.

30. CUBSCenter for Unified Biometrics and Sensors, NY, USA 30 ER 2 : Intuitive Similarity Measure Traditional Linear Regression Similarity: 91% Similarity: 31%

31. CUBSCenter for Unified Biometrics and Sensors, NY, USA 31 ER 2 : Intuitive Similarity Measure Linear Regression Given two sequences X=(x 1,x 2, …, x n ), Y=(y 1,y 2, …, y n ), then the similarity by R 2 of X and Y is: R 2 named R-squared because R 2 = (r) 2, where r is Pearson ’ s correlation r.

32. CUBSCenter for Unified Biometrics and Sensors, NY, USA 32 ER 2 : Intuitive Similarity Measure Extended Regression Traditional regression handles two 1-dimentional sequences. We extend it to multi-dimensional sequences as follows: We name it ER 2 since is an extension from 1-D to multi-D

33. CUBSCenter for Unified Biometrics and Sensors, NY, USA 33 ER 2 : Intuitive Similarity Measure The intuition of ER 2

34. CUBSCenter for Unified Biometrics and Sensors, NY, USA 34 ER 2 : Intuitive Similarity Measure Remarks on Linear Regression Advantages: Invariant to scale and translation; Similarity (Goodness-of-fit) makes sense. Disadvantages: One-one alignment, brittle. One-One alignmentDynamic alignment

35. CUBSCenter for Unified Biometrics and Sensors, NY, USA 35 ER 2 : Intuitive Similarity Measure We couple ER 2 with DTW-based Curve Matching Dynamic Alignment by DTW. However, we found direct DTW on two signatures is not very robust. We use Curve Matching, which is to calculate the total cost of changing one curve to fit another curve. The dynamic programming of DTW is used to realize the calculation.

36. CUBSCenter for Unified Biometrics and Sensors, NY, USA 36 ER 2 : Intuitive Similarity Measure  DTW-based Curve Matching Suppose we have two curves C and C ’. Curve matching is actually: Where speed(C) i = C i+1 -C i.

37. CUBSCenter for Unified Biometrics and Sensors, NY, USA 37 ER 2 : Intuitive Similarity Measure ER 2 coupled with Curve Matching The DTW warping path in the matrix is the path which has minimum average cumulative cost. The unmarked area is the constraint that path is allowed to go. ( y 2 is matched x 2, x 3, so we extend it to be two points in Y sequence.)

38. CUBSCenter for Unified Biometrics and Sensors, NY, USA 38 Experimental Results 1. Introduction- On-line signature verification √ 2. ER 2 : Intuitive Similarity Measure √ 3. Experimental Results 4. Demo – CUBS signature verification system 5. Conclusion 6. References

39. CUBSCenter for Unified Biometrics and Sensors, NY, USA 39 Experimental Results  Signature database The released signature datasets by SVC( First International Signature Verification Competition). SVC released the signatures of 80 individuals, 20 genuine and 20 skilled forgeries each.  Methods comparison ER 2 coupled with Curve Matching Vs. Curve Matching without ER 2

40. CUBSCenter for Unified Biometrics and Sensors, NY, USA 40 Experimental Results  Enrollment Enroll 6 genuine signatures from each individual.  Preprocessing Only X,Y-coordinates are used. Other information, such as Pressure, Altitude, Azimuth are not used in the experiments. 1) Smooth the raw sequence by Gaussian filter. 2) Rotate if necessary. 3) Normalize each signature by:

41. CUBSCenter for Unified Biometrics and Sensors, NY, USA 41 Experimental Results a) FRR and FAR of ER 2 (coupled with Curve Matching). b) FRR and FAR of Curve Matching (without ER 2 ). Both using universal threshold.

42. CUBSCenter for Unified Biometrics and Sensors, NY, USA 42 Experimental Results Table 2. EERs with universal or user-dependent threshold. Skilled forgeries are provided by the dataset, while random forgery means the forgeries are selected from the signatures of different individuals. *The results of SVC are available at http://www.cs.ust.hk/svc2004/results.html. We are team 14. http://www.cs.ust.hk/svc2004/results.html

43. CUBSCenter for Unified Biometrics and Sensors, NY, USA 43 Experimental Results  A project regarding on-line signature Recently, we have a Multimodal Biometrics project supported by US Army Laboratory. It requires to test signatures from 1000 individuals, each 2 as enrollment and 3 as queries. We collected 330 individuals so far. The preliminary ROC based on ER 2 is:

44. CUBSCenter for Unified Biometrics and Sensors, NY, USA 44 Experimental Results 1. Introduction - On-line signature verification √ 2. ER 2 : Intuitive Similarity Measure √ 3. Experimental Results √ 4. Demo – CUBS signature verification system 5. Conclusion 6. References

45. CUBSCenter for Unified Biometrics and Sensors, NY, USA 45

46. CUBSCenter for Unified Biometrics and Sensors, NY, USA 46 Demo – CUBS Sign. System

47. CUBSCenter for Unified Biometrics and Sensors, NY, USA 47 Conclusion We propose ER 2 as a similarity measure for multi- dimensional sequence matching. Signature verification system can use ER 2 coupled with curve matching for intuitive similarity output and higher performance as well. The experimental results are encouraging, although we have to notice that further evaluation on large and real databases is necessary. Our future work will explore the feasibility of ER 2 on dynamic features like pressure, speed, etc.

48. CUBSCenter for Unified Biometrics and Sensors, NY, USA 48 References [1] Rejean Plamondon, Guy Lorette. Automatic Signature Verification and Writer identification-the state of the art. Pattern Recognition, Vol.22, No.2, pp.107-131, 1989. [2] F. Leclerc and R. Plamondon. Automatic signature verification: the state of the art 1989-1993. International Journal of Pattern Recognition and Artificial Intelligence, 8(3):643-660, 1994. [3] Luan L. Lee, Toby Berger, Erez Aviczer. Reliable On-line Human Signature Verifications Systems. IEEE trans. On Pattern Analysis and Machine Intelligence, Vol. 18, No.6, June 1996. [4] R. Plamondon. The Design of On-line Signature Verification System: From Theory to Practice. Int ’ l J. Pattern Recognition and Artificial Intelligence, vol. 8, no. 3, pp. 795-811, 1994. [5] Mario E. Munich, Pietro Perona. Visual Identification by Signature Tracking. IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 25, No. 2, pp. 200-216, February 2003.

49. CUBSCenter for Unified Biometrics and Sensors, NY, USA 49 References [6] Vishvjit S. Nalwa. Automatic On-line Signature Verification. Proceedings of the IEEE, Vol. 85, No. 2, pp. 215-239, February 1997. [7] Jean-Jules Brault and Rejean Plamondon. Segmenting Hanwritten Signatures at Their Perceptually Important Points. IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol, 15, No. 9, pp. 953- 957, September 1993. [8] Taik H. Rhee, Sung J. Cho, Jin H. Kim. On-line Signature Verification Using Model-Guided Segmentation and Discriminative Feature Selection for Skilled Forgeries. Sixth International Conference on Document Analysis and Recognition (ICDAR '01), September, Seattle, Washington, 2001. [9] Thomas B. Sebastian, Philip N. Klein, Bejamin B. Kimia. On Aligning Curves. IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 25, No. 1, January 2003.

50. CUBSCenter for Unified Biometrics and Sensors, NY, USA 50 References [10] A.K. Jain, Friederike D. Griess and Scott D. Connell. On-line Signature Verification. Pattern Recognition, vol. 35, no. 12, pp. 2963--2972, Dec 2002. [11] K. Huang and H. Yan, “ On-Line Signature Verification Based on Dynamic segmentation and Global and Local Matching, ” Optical Eng., vol. 34, no. 12, pp. 3480-3487, 1995. [12] G. Lorette and R. Plamondon, “ Dynamic Approaches to Hand- written Signature Verification, ” Computer Processing of Hand-writing, pp. 21-47, 1990. [13] R. Martens and L. Claesen, “ On-Line Signature Verification by Dynamic Time-Warping, ” Proc. 13th Int ’ l Conf. Pattern Recognition, pp. 38-42, 1996. [14] B. Wirtz, “ Stroke-Based Time Warping for Signature Verification, ” Proc. Int ’ l Conf. Document Analysis and Recognition, pp. 179-182, 1995.