What Can Static Handwriting Features Reveal About Movement Dynamics? Hans-Leo Teulings, PhD NeuroScript, LLC Heidi H. Harralson, MA, D-BFDE Spectrum Forensic Int’l, LLC AFDE 2010 October 15, 2010, 1-3pm Embassy Suites Hotel, Phoenix, AZ

AFDE 2010, October 15, 2010, Scottsdale, AZ Info: ___________________________________________________________________http:// What Can Static Handwriting Features Reveal About Movement Dynamics? Hans-Leo Teulings, PhD, Heidi H. Harralson, MA, D-BFDE Most of the casework in Forensic Document Examination (FDE) involves the analysis of static handwriting samples on paper. However, the ability to reconstruct the dynamics of handwriting movement would help in distinguishing genuine writing from forged or disguised writing. Examining handwriting movement may also aid in revealing handedness, fatigue, medical condition, or the use of some medications. In this presentation, we will explain how shape features from scanned signature images can be used to draw inferences on the movement dynamics. In a practical demonstration, MovAlyzeR software will be used to extract static and dynamic information from a scanned signature image that can be compared to exemplars collected online from suspects. In a second study, we will show how electronic signature capture affects signatures due to the unusual stylus and stylus friction, size of the digital pad, unusual writing posture, and limited pad resolution. These issues can present challenges to FDEs when comparing electronic or point-of sale signatures with conventional ink signatures. Case examples will be discussed.

Disclosures Hans-Leo Teulings: Employed at NeuroScript that produces and markets MovAlyzeR handwriting analysis software. Heidi Harralson: None

Original Ink-on-Paper Signatures

Digitized Representations Original Ink-on-Paper Signatures

Contents Part I – Movement Dynamics Part II – E-Signature Procedures & Analysis

Part I: Movement Dynamics Each step (  ) may cause data degradation.  How to interpret electronic signatures?  Can movement be recovered from image?  Image: Pen on paper  Scan  Show  Recovery?   Movement: Special pen  Record  Show (Statics, Dynamics)

Equipment  MS PowerPoint  MS Paint –Visioneer Roadwarrior 120  NeuroScript MovAlyzeR + Externals Apps –Wacom Intuos3 + inking pen

Part I: Contents 1. Images 2. Movement 3. Filtering 4. Shape  Speed 5. Degradation: Sampling frequency 6. Degradation: Equidistant sample points 7. Forgery concealed by degradation 8. Degradation: Quantization of samples

1. Image Analysis Optical scan 600 dpi (scanner poor?) 8 bits gray scale Ballpoint 0.5 cm high Loops + Reversals Short + Long

Digitization/Quantization/Degradation: Pixelation 600 dpi  150 dpi Optical scan 150 dpi (So 28 pixels = 0.5 cm) 8 bits gray scale

Digitization/Quantization/Degradation: Bit depth 8 bits  1 bit Optical scan 600 dpi 1 bit gray scale (B/W)

Digitization/Quantization/Degradation: Bit depth & Pixelation Optical scan 150 dpi 1 bit gray scale (B/W)

Image Degradation: Valuable forensic information lost

Gray-scale image How MovAlyzeR likes it Optical scan 600 dpi 8 bits gray scale

Optical scan + Processed image Using MovAlyzeR image processing

Processed image  B/W Threshold  Skeleton  Width (Color Coded)  Segment at Forks  Segments

Processed image + Features for each Segment Seg- ment Ave- rage- Thick- ness Hori- zontal Start Ver- tical- Start Hori- zonal- Size Ver- tical- Size Trace- Lengt hSlant Straig htness -Rel- Error Loop- Area Arch- ed- ness Gar- land- Rel- Size Arcad e-Rel- Size Arch- edFre- quen- cy Gar- land- Fre- quen- cy Arcad eFre- quen- cy

B/W Threshold + Skeleton  = Fork point, separating 2 segments

Skeleton Future: Estimation of segment sequence Units in pixels (600=2.54 cm)

Skeleton Sequence of pixels (600 dpi)

Skeleton

Skeleton + Optical scan Matches image

Sources of variability + Motor-program noise + Digital equipment noise & quantization + Pen-on-surface friction noise + Inking noise (not discussed)

2. Pen movement recording Pen & Tablet (inking pen) Smartpen –Special paper –Normal paper (inking pen)

Inking Pen & Tablet Yields speed, pressure Normal papers Thin notepads

Smart Pen Clips also on thick notepads

Spectrum of large corpus of handwriting  Max motor speed:  5 loops/s (or 5 Hz)  Finger movement bandwidth <10 Hz  Noise bandwidth: Hz (Teulings, H.L., & Maarse, F.J. (1984). Digital recording and processing of handwriting movements. Human Movement Science, 3, ) Handwriting = Motor Program + Random Noise

Example: Slow Movement For example, 0.1 loop / s (=0.1 Hz)  Greater noise bandwidth Hz 5 loops / s (raw data) 0.1 loop / s (raw data)

Raw digitizer movement data = Sequence of samples 1 Sample = x: Horizontal pen-tip position, y: Vertical pen-tip position, z: Axial pen pressure This Exemplar: Small (0.5 cm) Fast (0.1 s / stroke) Light (50 g)

Raw digitizer data Sampled at 200 Hz constant frequency  Pen speed  Slowing in curves (More samples)

Raw digitizer samples SampleXYPressure “7 3““ ““33 6““44 7““55 8““67 9“ ““93 11““105 12““110 13““113 14““114 15““ “ ““ “ “ x, y, axial pressure, pen tilt Repeating samples  Pen stands still. Sample distance  Pen speed

x, y, axial pressure, tilt Pressure Data  Grams Raw digitizer samples SampleXYPressure “7 3““ ““33 6““44 7““55 8““67 9“ ““93 11““105 12““110 13““113 14““114 15““ “ ““ “ “105...

Raw digitizer samples Samples as points

Raw digitizer data Samples connected

Raw digitizer data + Optical Scan Samples match image Differences: Ink dispensing, pen tilt changes

Optical Scan 600 dpi, 8 bits gray scale (scanning resolution, but what about scanning accuracy or sharpness?)

Raw digitizer data 2540 dpi resolution, 10 bits pen pressure (Newer recordings: 5080 dpi, 11 bits pressure) (Pen position accuracy=250 dpi)

Digitizer data + Imagine skeleton (from Image analysis) Differences:  Ink dispensing  Pen tilt changes Image: 600 dpi = 600 pixels per 2.54 cm  1 pixel = cm

Image Skeleton Like samples but no speed information

3. Low-pass Movement Filtering Frequency Spectrum = Signal + Noise Suppress: Noise (= flat part above10 Hz). Gradual cutoff: 4-16 Hz

Raw samples From digitizer

Raw samples + Filtered samples Filtered at 10 Hz Match except higher-frequency irregularities

Filtered samples Segmented into Up and Down strokes

Different filter frequencies Filtering out more essential frequencies, yields stronger distortions.

Raw Data Raw data from from inking pen

200 Hz filter Raw + Filtered

20 Hz filter

Raw + Filtered 14 Hz filter

Raw + Filtered 10 Hz filter

Raw + Filtered 7 Hz filter

Raw + Filtered 5 Hz filter

Raw + Filtered 4 Hz filter

Raw + Filtered 3 Hz Filter

Raw + Filtered 5 Hz filter + Rescaling Vertical Component

Raw + Filtered 4 Hz filter + Rescaling Vertical Component

Raw + Filtered 3 Hz Filter + Rescaling Vertical Component

Effects of filtering Filtering at 5 Hz distorts handwriting Filtering at 10 Hz removes equipment and friction noise

 Driving an automobile safer: Slow down in curves (  Go faster on straight parts)  Make faster U-turn: Sharper curve (  Faster reversal in sharper curves) Likewise for a pen movement on paper! 4. Shape  Speed

Shape  Speed Curve Radius R R.6 cm

Shape  Speed “Pen moves faster when R greater” Curve Radius (cm) versus Time (s) R(cm) R.6 cm

Shape  Speed “Pen moves faster when R greater” Curve Radius (cm) versus Time (s)  Pen Speed (cm/s) R.6 cm R(cm)

“Faster reversal in sharper curves” Curvature (1/cm) =1/Curve Radius  Angular Velocity in “straight” parts =12.5 rad / s =2 loops / s = Shape  Speed

Shape yields speed estimate Pen moves: Faster in straighter strokes Slower in sharper curves

Degraded Pen Movement Data 5. Low sampling frequency 6. Equidistant resampling 7. Forgery attempt 8. Quantization

5. Can we recover movement? From only 29 samples at fixed 25 Hz? Erratum: x, y scales show half the actual size from here on.

Experiment 1. Movement sampled at 200 Hz 2. Keep only 1 of 8 samples (=Downsample to 25 Hz) 3. Upsample 8x  Can original be restored?

Handwriting of bandwidth10 Hz requires only 20 samples / second 1 Loop (=1 Cycle =2 Samples) (Sampling Theorem, or Cardinal Theorem of Interpolation Theory by Nyquist, Whittaker, Kotelnikov, or Shannon).

Sampled at 200 Hz Raw data from pen tablet

Downsampled to 25 Hz From 200 Hz by selecting 1 of 8 samples

Downsample to 25 Hz These 29 samples (1.2 s) are sufficient! (24 samples at 20 Hz would even do)

Fluent Handwriting Described by >2 samples per stroke

Fluent Handwriting Described by >2 samples per stroke

Sampled at 25 Hz (downsampled from original 200 Hz)

Fluent Handwriting Filtered at 10 Hz

Fluent Handwriting Filtered at 10 Hz

Add frequencies of amplitude 0

Compare original and upsampled Scan 200 Hz sampling   25 Hz sampling 8x Upsampling 

Original 200 Hz sampling

Original + Upsampled 200 Hz sampling 25 Hz upsampled 8x Almost perfect recovery.

Upsampled 25 Hz upsampled 8x

Scan + Upsampled

6. Samples at fixed distances Result of engineering, not forensic thinking

Resampled to fixed distances Resampled to 125 points/cm (= 300 dpi)

Compare: Fixed-frequency sampling Original raw data (generated by equipment)

Samples at fixed distances No repeating samples  No info on pen stops SampleXYPressure

Samples at fixed distances Speed ~ Curve Radius Speed = Constant

Fixed frequency vs Fixed distances Fixed frequency Fixed distance transf.  More high freqs

7. Forgery attempt Trace original Magnifying glass Non-dominant hand, 4th attempt (43 times slower)

Forgery (Frequency constant) Hard to forge under constant frequency Downsampled 43x

Forgery (Distances Constant) Easier to forge

Forgery vs Optical scan Traced with handicap (writing with non-dominant hand)

Optical scan vs Forgery Traced with handicap

Filtered vs Unfiltered (raw) forgery Downsampled 43x and filtered 43x lower (200Hz  4.6Hz, 10 Hz  2.3 Hz)

Filtered Unfiltered (raw) forgery

Filtered Forgery vs Optical scan Downsampled 43x and filtered 43x lower (200Hz  4.6Hz, 10 Hz  2.3 Hz)

Optical scan vs Filtered Forgery

8. Quantization Noise Image Pixelation: (actual) Digitizer quantization: (artificially exaggerated)

Raw samples From the digitizer

Quantization noise artificially increased 32x (i.e., resolution reduced by 5 bits) Raw + Quantized samples

Quantized samples Quantization noise artificially increased 32x

Quantized samples SampleXYPressure ““ ““ “115 36“ “117 38“ ““ “ “119 44“ “137 46““170 47“ ““ Looks like equidistant samples! Repeated samples  Speed

Quantized samples + Optical scan

Quantized samples

Frequency Spectra Original data Artificially quantized data  High-frequency noise added  Removed by filtering

Quantized samples AFTER filtering Using standard 10 Hz filter

Quantized, filtered samples + Filtered samples No large differences

Part II: E-Signature Procedures & Analysis 1. Digital 2. Static 3. Dynamic/Biometric

Digitized SignaturesPaper/Ink Signatures

Examination Process Identification: E-signature type 1. Digital 2. Static 3. Dynamic/Biometric Analysis Limitations Opinion

1. Digital Capturing Method –Mathematical, cryptographic –No handwriting image or data collected Analysis Not within scope of DE casework Refer to forensic computer expert Limitations –NA Opinion –NA

2. Static Capturing Method –Digital pad/device –Printed or electronic handwriting image Why is this static?

Analysis Electronic capture device/software –Verify no dynamic data captured –Limitations with technology Image –Resolution and digitization –Use MovAlyzeR to analyze image Comparable standards –Comparison of ink signatures to static electronic signatures has considerable limitations –Capture request standards using similar conditions

Limitations - Technology Tablet resolution Display/printer resolution Digital device unable to capture full image Data transmission loss Lack of grayscale No pressure indicators No speed indicators Limitations in capturing high writing speeds, airstrokes, heavy pressure

Data Transmission Loss It was found that even moderate amounts of loss can lead to serious degradation in error rates in on-line signature verification systems It is important that distributed on-line signature verification systems ensure that packets are safely delivered Richiardi, Fierrez-Aguilar, Ortega-Garcia, Drygajlo. (2004). On-line signature verification resilience to packet loss in IP networks. 2 nd Workshop on Biometrics on the Internet.

Limitations - Signer Delayed feedback Barriers/boxes “Annoying” instructions Unsupported hand Writing tablet surface Broad-tipped writing instrument Unusual “pens” Awkward position Adapting to the device and associated conditions can alter natural motor program

Practical Experiment Using commercially available electronic signature capturing software, signatures were captured using different “pens”: –Inking tablet pen –Tablet pen –Touch pad –Mouse

The digital interpretation of the ink signature shows inconsistencies in the way the lighter “airstrokes” are represented.

Tablet Pen Samples-left: Using a tablet pen, writer is making adjustments in order to fit into tablet box (signature ending). Pen lifts have decreased. Ink signature

Inking PenTablet Pen

Inking PenTablet Pen

Inking PenTablet Pen

Inking PenTablet Pen

Touch Pad Samples-left: Writer is “drawing” the signature using a touch pad instead of writing it due to the awkward writing “instrument” Ink signature

Mouse Samples-left: Writer is “drawing” the signature instead of writing it with a mouse due to the awkward writing “instrument”. The mouse further increases awkwardness in comparison to the touch pad. Ink signature

Opinion Poor resolution images + incomparable standards = weak or inconclusive opinion High resolution images + comparable standards = qualified opinion Conclusive opinions may not be possible with static images

3. Dynamic/Biometric Capturing Method –Digital pad/device –Software Handwriting image Dynamic/biometric data

Analysis Electronic capture device/software –Limitations with technology Image –Resolution and digitization –Correspond w/dynamic data Dynamic/biometric data –Feature analysis Comparable standards –Do they exist?

Biometric Data Feature extraction varies considerably between software systems –Speed –Acceleration –Pressure –X/Y coordinates Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

Available Systems Cyber-SIGN: speed, shape, pressure, strokes (including airstrokes) KeCrypt: speed, pressure Topaz: segment timing, signature speed BioSig-ID –No hardware required…sign or draw symbol with a mouse –Captures speed, direction, length

Consistent Features Lei & Govindaraju (2005) examined 22 commonly used features for consistency and discriminative power…the most consistent features were: –Coordinate sequence (X,Y, X/Y) –Speed –Angle (X axis) Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

Consistent Features Pressure is a commonly used feature, but the mean consistency was not high. Large variation in pressure does not necessarily show a forgery, but a very similar pressure pattern is a strong indication of a genuine signature Lei & Govindaraju. (2005). A comparative study on the consistency of features in on-line signature verification. Pattern Recognition Letters, 26(15).

Kinematics Research Phillips, Muller, & Ogeil. (2007). Alcohol intoxication and handwriting: A kinematic analysis. IGS Proceedings. –Disturbances in length; shortened duration of decelerative phase; more spectral power around 4Hz Mohammed, Found, Caligiuri, & Rogers. (2009). Pen pressure as a discriminatory feature between genuine and forged signatures. IGS Proceedings. –Genuine had higher pen pressure; forgeries had higher pen pressure variability Harralson, Teulings, & Farley. (2009). Handwriting variability in movement disorder patients and effects of fatigue. IGS Proceedings. –Patients had slower writing speed and/or increased variability than healthy controls

Result 4: Normalized Jerk (in Condition SEN) increases with dosage of Risperidone in schizophrenia patients while the conventional SAS, AIMS, and BAS tests show no differences. SAS AIMS BAS Michael P. Caligiuri, Hans-Leo Teulings, Charles E. Dean3, Alexander B. Niculescu, James B. Lohr Handwriting Movement Analysis to Monitor Drug-Induced Movement Side Effects in Schizophrenia Patients AzBioExpo2007, 20 June 2007, Phoenix Convention Center, Arizona BioIndustry Association & Flinn

Frequency Spectrum Essential Tremor Genuine Essential Tremor Forgery Harralson, Teulings, Farley. (2007). Comparison of handwriting kinematics in movement disorders and forgery. IGS Proceedings.

Opinion In sophisticated e-signature capturing systems, biometric data can be more valuable than a wet ink signature…this can lead to strong opinions…if comparable e-signature standards are available Many systems are not sophisticated and only capture a limited number of biometric parameters in an insufficient way…leads to inconclusive or weak opinions

Training Changing paradigm of handwriting technology requires new FDE training –Wall Street Journal article: “How Handwriting Changes the Brain” Literature – –IGS Proceedings Resources and Tools – –MovAlyzeR (free 15 day trial) News articles –How Handwriting Trains the Brain

Contacts Hans-Leo Teulings, PhD –NeuroScript, LLC – Heidi H. Harralson, MA, D-BFDE –Spectrum Forensic International, LLC