1. What Can Static Handwriting Features Reveal About Movement Dynamics? Hans-Leo Teulings, PhD NeuroScript, LLC Heidi H. Harralson, MA, D-BFDE Spectrum Forensic Int’l, LLC AFDE 2010 October 15, 2010, 1-3pm Embassy Suites Hotel, Phoenix, AZ

2. AFDE 2010, October 15, 2010, Scottsdale, AZ Info: http://www.afde.org/symposium.html ___________________________________________________________________http://www.afde.org/symposium.html What Can Static Handwriting Features Reveal About Movement Dynamics? Hans-Leo Teulings, PhD, Heidi H. Harralson, MA, D-BFDE Most of the casework in Forensic Document Examination (FDE) involves the analysis of static handwriting samples on paper. However, the ability to reconstruct the dynamics of handwriting movement would help in distinguishing genuine writing from forged or disguised writing. Examining handwriting movement may also aid in revealing handedness, fatigue, medical condition, or the use of some medications. In this presentation, we will explain how shape features from scanned signature images can be used to draw inferences on the movement dynamics. In a practical demonstration, MovAlyzeR software will be used to extract static and dynamic information from a scanned signature image that can be compared to exemplars collected online from suspects. In a second study, we will show how electronic signature capture affects signatures due to the unusual stylus and stylus friction, size of the digital pad, unusual writing posture, and limited pad resolution. These issues can present challenges to FDEs when comparing electronic or point-of sale signatures with conventional ink signatures. Case examples will be discussed.

3. Disclosures Hans-Leo Teulings: Employed at NeuroScript that produces and markets MovAlyzeR handwriting analysis software. Heidi Harralson: None

4. Original Ink-on-Paper Signatures

5. Digitized Representations Original Ink-on-Paper Signatures

6. Contents Part I – Movement Dynamics Part II – E-Signature Procedures & Analysis

7. Part I: Movement Dynamics Each step (  ) may cause data degradation.  How to interpret electronic signatures?  Can movement be recovered from image?  Image: Pen on paper  Scan  Show  Recovery?   Movement: Special pen  Record  Show (Statics, Dynamics)

8. Equipment  MS PowerPoint  MS Paint –Visioneer Roadwarrior 120  NeuroScript MovAlyzeR + Externals Apps –Wacom Intuos3 + inking pen

9. Part I: Contents 1. Images 2. Movement 3. Filtering 4. Shape  Speed 5. Degradation: Sampling frequency 6. Degradation: Equidistant sample points 7. Forgery concealed by degradation 8. Degradation: Quantization of samples

10. 1. Image Analysis Optical scan 600 dpi (scanner poor?) 8 bits gray scale Ballpoint 0.5 cm high Loops + Reversals Short + Long

11. Digitization/Quantization/Degradation: Pixelation 600 dpi  150 dpi Optical scan 150 dpi (So 28 pixels = 0.5 cm) 8 bits gray scale

12. Digitization/Quantization/Degradation: Bit depth 8 bits  1 bit Optical scan 600 dpi 1 bit gray scale (B/W)

13. Digitization/Quantization/Degradation: Bit depth & Pixelation Optical scan 150 dpi 1 bit gray scale (B/W)

14. Image Degradation: Valuable forensic information lost

15. Gray-scale image How MovAlyzeR likes it Optical scan 600 dpi 8 bits gray scale

16. Optical scan + Processed image Using MovAlyzeR image processing

17. Processed image  B/W Threshold  Skeleton  Width (Color Coded)  Segment at Forks  Segments

18. Processed image + Features for each Segment Seg- ment Ave- rage- Thick- ness Hori- zontal Start Ver- tical- Start Hori- zonal- Size Ver- tical- Size Trace- Lengt hSlant Straig htness -Rel- Error Loop- Area Arch- ed- ness Gar- land- Rel- Size Arcad e-Rel- Size Arch- edFre- quen- cy Gar- land- Fre- quen- cy Arcad eFre- quen- cy 10.95391350.7941.32 00.0520.187000000 21.97431321.595.296.6100.14200.837 0110 31.87855418.54.2319.130.60.6910000000 41.79915120.47.9424.633.41.640000000 52.3889520.5290.2651.0600.1320.265000000 61.9612311810.61824.602.170-10.20 101 72.46137838.730.794921.60.1290000000 81.8612512211.47.4112.7-12.30.5380000000 91.941781328.25.5610.100.2370-2.450 101 102.121871240.5291.061.5900.0780000000 111.911841280.2651.061.3200.0730.324000000 122.36201957.141.067.4127.40.0840000000 132.082561188.215.621.201.650-8.110 101 142.652591111.320.2651.5931.50.0670.374000000 151.6825910524.68.4750.815.12.8511.8000000 162.012881127.94 15.301.820-6.430 101

19. B/W Threshold + Skeleton  = Fork point, separating 2 segments

20. Skeleton Future: Estimation of segment sequence Units in pixels (600=2.54 cm)

21. Skeleton Sequence of pixels (600 dpi)

22. Skeleton

23. Skeleton + Optical scan Matches image

24. Sources of variability + Motor-program noise + Digital equipment noise & quantization + Pen-on-surface friction noise + Inking noise (not discussed)

25. 2. Pen movement recording Pen & Tablet (inking pen) Smartpen –Special paper –Normal paper (inking pen)

26. Inking Pen & Tablet Yields speed, pressure Normal papers Thin notepads

27. Smart Pen Clips also on thick notepads

28. Spectrum of large corpus of handwriting  Max motor speed:  5 loops/s (or 5 Hz)  Finger movement bandwidth <10 Hz  Noise bandwidth: 5 - 10 Hz (Teulings, H.L., & Maarse, F.J. (1984). Digital recording and processing of handwriting movements. Human Movement Science, 3, 193-217.) Handwriting = Motor Program + Random Noise

29. Example: Slow Movement For example, 0.1 loop / s (=0.1 Hz)  Greater noise bandwidth 0.1 - 10 Hz 5 loops / s (raw data) 0.1 loop / s (raw data)

30. Raw digitizer movement data = Sequence of samples 1 Sample = x: Horizontal pen-tip position, y: Vertical pen-tip position, z: Axial pen pressure This Exemplar: Small (0.5 cm) Fast (0.1 s / stroke) Light (50 g)

31. Raw digitizer data Sampled at 200 Hz constant frequency  Pen speed  Slowing in curves (More samples)

32. Raw digitizer samples SampleXYPressure 12620112192 22611“7 3““13 426061121621 5““33 6““44 7““55 8““67 9“1121279 10““93 11““105 12““110 13““113 14““114 15““112 162610“111 17261411208110 182620““ 192625“109 202631“105... x, y, axial pressure, pen tilt Repeating samples  Pen stands still. Sample distance  Pen speed

33. x, y, axial pressure, tilt Pressure Data  Grams Raw digitizer samples SampleXYPressure 12620112192 22611“7 3““13 426061121621 5““33 6““44 7““55 8““67 9“1121279 10““93 11““105 12““110 13““113 14““114 15““112 162610“111 17261411208110 182620““ 192625“109 202631“105...

34. Raw digitizer samples Samples as points

35. Raw digitizer data Samples connected

36. Raw digitizer data + Optical Scan Samples match image Differences: Ink dispensing, pen tilt changes

37. Optical Scan 600 dpi, 8 bits gray scale (scanning resolution, but what about scanning accuracy or sharpness?)

38. Raw digitizer data 2540 dpi resolution, 10 bits pen pressure (Newer recordings: 5080 dpi, 11 bits pressure) (Pen position accuracy=250 dpi)

39. Digitizer data + Imagine skeleton (from Image analysis) Differences:  Ink dispensing  Pen tilt changes Image: 600 dpi = 600 pixels per 2.54 cm  1 pixel = 0.004 cm

40. Image Skeleton Like samples but no speed information

41. 3. Low-pass Movement Filtering Frequency Spectrum = Signal + Noise Suppress: Noise (= flat part above10 Hz). Gradual cutoff: 4-16 Hz

42. Raw samples From digitizer

43. Raw samples + Filtered samples Filtered at 10 Hz Match except higher-frequency irregularities

44. Filtered samples Segmented into Up and Down strokes

45. Different filter frequencies Filtering out more essential frequencies, yields stronger distortions.

46. Raw Data Raw data from from inking pen

47. 200 Hz filter Raw + Filtered

48. 20 Hz filter

49. Raw + Filtered 14 Hz filter

50. Raw + Filtered 10 Hz filter

51. Raw + Filtered 7 Hz filter

52. Raw + Filtered 5 Hz filter

53. Raw + Filtered 4 Hz filter

54. Raw + Filtered 3 Hz Filter

55. Raw + Filtered 5 Hz filter + Rescaling Vertical Component

56. Raw + Filtered 4 Hz filter + Rescaling Vertical Component

57. Raw + Filtered 3 Hz Filter + Rescaling Vertical Component

58. Effects of filtering Filtering at 5 Hz distorts handwriting Filtering at 10 Hz removes equipment and friction noise

59.  Driving an automobile safer: Slow down in curves (  Go faster on straight parts)  Make faster U-turn: Sharper curve (  Faster reversal in sharper curves) Likewise for a pen movement on paper! 4. Shape  Speed

60. Shape  Speed Curve Radius R R.6 cm

61. Shape  Speed “Pen moves faster when R greater” Curve Radius (cm) versus Time (s) R(cm) R.6 cm

62. Shape  Speed “Pen moves faster when R greater” Curve Radius (cm) versus Time (s)  Pen Speed (cm/s) R.6 cm R(cm)

63. “Faster reversal in sharper curves” Curvature (1/cm) =1/Curve Radius  Angular Velocity in “straight” parts =12.5 rad / s =2 loops / s = Shape  Speed

64. Shape yields speed estimate Pen moves: Faster in straighter strokes Slower in sharper curves

65. Degraded Pen Movement Data 5. Low sampling frequency 6. Equidistant resampling 7. Forgery attempt 8. Quantization

66. 5. Can we recover movement? From only 29 samples at fixed 25 Hz? Erratum: x, y scales show half the actual size from here on.

67. Experiment 1. Movement sampled at 200 Hz 2. Keep only 1 of 8 samples (=Downsample to 25 Hz) 3. Upsample 8x  Can original be restored?

68. Handwriting of bandwidth10 Hz...... requires only 20 samples / second 1 Loop (=1 Cycle =2 Samples) (Sampling Theorem, or Cardinal Theorem of Interpolation Theory by Nyquist, Whittaker, Kotelnikov, or Shannon).

69. Sampled at 200 Hz Raw data from pen tablet

70. Downsampled to 25 Hz From 200 Hz by selecting 1 of 8 samples

71. Downsample to 25 Hz These 29 samples (1.2 s) are sufficient! (24 samples at 20 Hz would even do)

72. Fluent Handwriting Described by >2 samples per stroke

73. Fluent Handwriting Described by >2 samples per stroke

74. Sampled at 25 Hz (downsampled from original 200 Hz)

75. Fluent Handwriting Filtered at 10 Hz

76. Fluent Handwriting Filtered at 10 Hz

77. Add frequencies of amplitude 0

78. Compare original and upsampled Scan 200 Hz sampling   25 Hz sampling 8x Upsampling 

79. Original 200 Hz sampling

80. Original + Upsampled 200 Hz sampling 25 Hz upsampled 8x Almost perfect recovery.

81. Upsampled 25 Hz upsampled 8x

82. Scan + Upsampled

83. 6. Samples at fixed distances Result of engineering, not forensic thinking

84. Resampled to fixed distances Resampled to 125 points/cm (= 300 dpi)

85. Compare: Fixed-frequency sampling Original raw data (generated by equipment)

86. Samples at fixed distances No repeating samples  No info on pen stops SampleXYPressure 12620112192 226061121373 3262111208109 4263911208108 5265711208113 6267411213115 7269111217115 8270811222113 9272511228112 10274111236112...

87. Samples at fixed distances Speed ~ Curve Radius Speed = Constant

88. Fixed frequency vs Fixed distances Fixed frequency Fixed distance transf.  More high freqs

89. 7. Forgery attempt Trace original Magnifying glass Non-dominant hand, 4th attempt (43 times slower)

90. Forgery (Frequency constant) Hard to forge under constant frequency Downsampled 43x

91. Forgery (Distances Constant) Easier to forge

92. Forgery vs Optical scan Traced with handicap (writing with non-dominant hand)

93. Optical scan vs Forgery Traced with handicap

94. Filtered vs Unfiltered (raw) forgery Downsampled 43x and filtered 43x lower (200Hz  4.6Hz, 10 Hz  2.3 Hz)

95. Filtered Unfiltered (raw) forgery

96. Filtered Forgery vs Optical scan Downsampled 43x and filtered 43x lower (200Hz  4.6Hz, 10 Hz  2.3 Hz)

97. Optical scan vs Filtered Forgery

98. 8. Quantization Noise Image Pixelation: (actual) Digitizer quantization: (artificially exaggerated)

99. Raw samples From the digitizer

100. Quantization noise artificially increased 32x (i.e., resolution reduced by 5 bits) Raw + Quantized samples

101. Quantized samples Quantization noise artificially increased 32x

102. Quantized samples SampleXYPressure 29276811248112 30280011280114 31““112 32283211312112 33““113 34286411344112 352896“115 36“11376114 372928“117 38“11408115 39296011440119 40““118 41299211472118 42“11504118 432992“119 44“11536125 452992“137 46““170 47“11568202 48““212... Looks like equidistant samples! Repeated samples  Speed

103. Quantized samples + Optical scan

104. Quantized samples

105. Frequency Spectra Original data Artificially quantized data  High-frequency noise added  Removed by filtering

106. Quantized samples AFTER filtering Using standard 10 Hz filter

107. Quantized, filtered samples + Filtered samples No large differences

108. Part II: E-Signature Procedures & Analysis 1. Digital 2. Static 3. Dynamic/Biometric

109. Digitized SignaturesPaper/Ink Signatures

110. Examination Process Identification: E-signature type 1. Digital 2. Static 3. Dynamic/Biometric Analysis Limitations Opinion

111. 1. Digital Capturing Method –Mathematical, cryptographic –No handwriting image or data collected Analysis Not within scope of DE casework Refer to forensic computer expert Limitations –NA Opinion –NA

112. 2. Static Capturing Method –Digital pad/device –Printed or electronic handwriting image Why is this static?

114. Analysis Electronic capture device/software –Verify no dynamic data captured –Limitations with technology Image –Resolution and digitization –Use MovAlyzeR to analyze image Comparable standards –Comparison of ink signatures to static electronic signatures has considerable limitations –Capture request standards using similar conditions

115. Limitations - Technology Tablet resolution Display/printer resolution Digital device unable to capture full image Data transmission loss Lack of grayscale No pressure indicators No speed indicators Limitations in capturing high writing speeds, airstrokes, heavy pressure

117. Data Transmission Loss It was found that even moderate amounts of loss can lead to serious degradation in error rates in on-line signature verification systems It is important that distributed on-line signature verification systems ensure that packets are safely delivered Richiardi, Fierrez-Aguilar, Ortega-Garcia, Drygajlo. (2004). On-line signature verification resilience to packet loss in IP networks. 2 nd Workshop on Biometrics on the Internet.

118. Limitations - Signer Delayed feedback Barriers/boxes “Annoying” instructions Unsupported hand Writing tablet surface Broad-tipped writing instrument Unusual “pens” Awkward position Adapting to the device and associated conditions can alter natural motor program

119. Practical Experiment Using commercially available electronic signature capturing software, signatures were captured using different “pens”: –Inking tablet pen –Tablet pen –Touch pad –Mouse www.ez-signature.com

120. The digital interpretation of the ink signature shows inconsistencies in the way the lighter “airstrokes” are represented.

125. Tablet Pen Samples-left: Using a tablet pen, writer is making adjustments in order to fit into tablet box (signature ending). Pen lifts have decreased. Ink signature

126. Inking PenTablet Pen

127. Inking PenTablet Pen

128. Inking PenTablet Pen

129. Inking PenTablet Pen

130. Touch Pad Samples-left: Writer is “drawing” the signature using a touch pad instead of writing it due to the awkward writing “instrument” Ink signature

131. Mouse Samples-left: Writer is “drawing” the signature instead of writing it with a mouse due to the awkward writing “instrument”. The mouse further increases awkwardness in comparison to the touch pad. Ink signature

132. Opinion Poor resolution images + incomparable standards = weak or inconclusive opinion High resolution images + comparable standards = qualified opinion Conclusive opinions may not be possible with static images

133. 3. Dynamic/Biometric Capturing Method –Digital pad/device –Software Handwriting image Dynamic/biometric data

134. Analysis Electronic capture device/software –Limitations with technology Image –Resolution and digitization –Correspond w/dynamic data Dynamic/biometric data –Feature analysis Comparable standards –Do they exist?

135. Biometric Data Feature extraction varies considerably between software systems –Speed –Acceleration –Pressure –X/Y coordinates Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

136. Available Systems Cyber-SIGN: speed, shape, pressure, strokes (including airstrokes) KeCrypt: speed, pressure Topaz: segment timing, signature speed BioSig-ID –No hardware required…sign or draw symbol with a mouse –Captures speed, direction, length

137. Consistent Features Lei & Govindaraju (2005) examined 22 commonly used features for consistency and discriminative power…the most consistent features were: –Coordinate sequence (X,Y, X/Y) –Speed –Angle (X axis) Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

138. Consistent Features Pressure is a commonly used feature, but the mean consistency was not high. Large variation in pressure does not necessarily show a forgery, but a very similar pressure pattern is a strong indication of a genuine signature Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

139. Kinematics Research Phillips, Muller, & Ogeil. (2007). Alcohol intoxication and handwriting: A kinematic analysis. IGS Proceedings. –Disturbances in length; shortened duration of decelerative phase; more spectral power around 4Hz Mohammed, Found, Caligiuri, & Rogers. (2009). Pen pressure as a discriminatory feature between genuine and forged signatures. IGS Proceedings. –Genuine had higher pen pressure; forgeries had higher pen pressure variability Harralson, Teulings, & Farley. (2009). Handwriting variability in movement disorder patients and effects of fatigue. IGS Proceedings. –Patients had slower writing speed and/or increased variability than healthy controls

140. Result 4: Normalized Jerk (in Condition SEN) increases with dosage of Risperidone in schizophrenia patients while the conventional SAS, AIMS, and BAS tests show no differences. SAS AIMS BAS Michael P. Caligiuri, Hans-Leo Teulings, Charles E. Dean3, Alexander B. Niculescu, James B. Lohr Handwriting Movement Analysis to Monitor Drug-Induced Movement Side Effects in Schizophrenia Patients AzBioExpo2007, 20 June 2007, Phoenix Convention Center, Arizona BioIndustry Association & Flinn

141. Frequency Spectrum Essential Tremor Genuine Essential Tremor Forgery Harralson, Teulings, Farley. (2007). Comparison of handwriting kinematics in movement disorders and forgery. IGS Proceedings.

142. Opinion In sophisticated e-signature capturing systems, biometric data can be more valuable than a wet ink signature…this can lead to strong opinions…if comparable e-signature standards are available Many systems are not sophisticated and only capture a limited number of biometric parameters in an insufficient way…leads to inconclusive or weak opinions

143. Training Changing paradigm of handwriting technology requires new FDE training –Wall Street Journal article: “How Handwriting Changes the Brain” Literature –www.graphonomics.orgwww.graphonomics.org –IGS Proceedings Resources and Tools –www.neuroscript.netwww.neuroscript.net –MovAlyzeR (free 15 day trial) News articles –How Handwriting Trains the Brain http://www.neuroscript.net/forum/showthread.php?3137 http://www.neuroscript.net/forum/showthread.php?3137

144. Contacts Hans-Leo Teulings, PhD –NeuroScript, LLC –www.neuroscript.netwww.neuroscript.net –hlteulings@neuroscriptsoftware.comhlteulings@neuroscriptsoftware.com Heidi H. Harralson, MA, D-BFDE –Spectrum Forensic International, LLC –Spectrum008@aol.com