Multiple Agents for Pattern Recognition Louis Vuurpijl

Contents The problem: handwriting recognition PR1: The traditional solution –(some) solutions at the NICI PR2: Multiple classifiers –(some) solutions at the NICI PR3: Could MAPR be a solution? –our current achievements

Sources of variation (Schomaker, Plamondon et al ’00) Affine transforms Style Variations Neuro- biomechanichal Variations Order Variability

The more writers, the more….

Is this a problem? Shrihari ICDAR’01: “No problem” Handwriting is individual, so can be used in court (as a fingerprint)! IWFHRxx, ICDAR, IJDAR... “Yes!” We have 99+ digit recognition, 98+ character recognition and 90+ for isolated words

Pattern Recognition (I) Raw data X(t),Y(t),P(t) Class labels

Pattern Recognition (I) Preprocessing Segmentation Feature Extraction Classification Class labels Raw data X(t),Y(t),P(t)

Solutions at the NICI More than 20 years of experience in –Handwriting production (Thomassen, van Galen, Meulenbroek, Maarse, Schomaker, et al) –Handwriting recognition (Schomaker, Teulings, Vuurpijl, et al) Keywords: –Use knowledge about human handwriting –Specialization and –Fusion

The basis of handwriting X(t),Y(t),P(t) Va(t)

UNIPEN data

Zooming in on writing styles UNIPEN styles: print, cursive, mixed

The lean recognition machine

Clustering on stroke- features (IWFHR’96) Specialization boosts recognition performance, while reducing computational and memory requirements

Finding structure in diversity (ICDAR’97)

Hierarchy in character shapes

Allograph prototypes

dScript: a MAPR system (’00)

dScript: 9 classifiers Neural networks (MLP, Kohonen) Nearest neighbour & clustering Structural/geometrical Support vector machines Hidden markov models Fusion through classifier combination and Multiple agents (IWFHR’98,’00,’02)

Pattern Recognition (II) Raw data Class labels Classifier 0 Classifier(i) Classifier Combination

Classifier combination (van Erp’00,’02)

Changing contexts static architecture - what if “Go 551” was intended?

Dynamic PR Through extra heuristic information

Dynamic PR Through extra features (add PENUP) Features determine how you look at data

Dynamic PR Knowing when to use which feature

Dynamic PR Knowing when to use which feature

Dynamic PR Knowing when to use which feature/algorithm Through a knowledge base of PR Through a library of PR modules Through negotiation protocols Knowing how to use which PR module

Pattern Recognition (III) Raw data Class labels MAPR

What is an agent? (Wooldridge and Jennings, 1995) A software system with: Goals: ``What do you want?'' or ``What can you do?'' –I can solve 0-6 conflicts Beliefs and reasoning: ``How do you realize this goal?'' –I solve this 0-6 conflict using modules PR1 and PR2 –and features #84 and #96, extracted by FE(i) and FE(j) Assertions with confidence: –Based on my experience and these features I belief this input is a ``6'' with confidence 0.9. –I have been correct in 90% of the cases in the past.

Agent framework

MAPR: Our goal A distributed intelligent agent framework, with PR modules, symbolic equivalents and PR language. Driven by problem constraints and with learning capabilities.

Our current MAPR system Distributed processing over internet using sockets. Interfaces to KQML and Jatlite. Agents know about the environment. Agents know about the available PR modules and data. Agents interact with other agents. Agents to detect problems and conflicts

A proof of concept Trained recognition system hclus –15557 digits from UNIPEN –7778 train, 7779 test (95.9%) , 7-2, 4-6 conflicts solved  97%

But This is all hard-wired if confidence top[0] too low then solve(top[0],top[1],...) solve(1,7,2) = best(1-7,1-2,7-2)

Current research on MAPR Knowledge base PR language & implementation PR negotiation mechanisms

Conclusions Online HWR is still unsolved MCS can improve recognition rates, but..... –Hard-wired PR modules –Examples where dynamic PR is needed MAPR is a new paradigm that exploits knowledge about when to use which features or algorithms But how to implement shared access to knowledge? And how to perform agent-like negotiations ?