1. Multimodal Input of Mathematical Symbols (Model 1)
Speech Input
Handwriting Input
HANDWRITING MODULE
SPEECH MODULE
Cscript code using Microsoft Speech SDK 5.1 speech.js
Speech Grammar math.xml
Handwriting Input using Microsoft Tablet PC SDK
Saves output to file
File testfile.txt
Polls file to look for changes
SPEECH WAIT talktome.lisp
Calls speech-event if there are new speech information arrives
Speech-event talktome.lisp
Time-synchronized Alternatively, the Speech Queue and Handwriting Queue can integrated into 1 queue
Inserts into queue with time information
SPEECH QUEUE *eventqueue*
HANDWRITING QUEUE
Multimodal Integrator mathinput.lisp
Feeds result to SKEME
SKEME skeme.lisp mathinput.lisp (and other supporting files)
Alternatively, mouse input and keyboard input can also be “fed” to Multimodal integrator.
Data file
Data flow
Input
Running Programs
Mouse input
Keyboard input

2. Multimodal Input of Mathematical Symbols (Model 2)
Speech Input
Handwriting Input
HANDWRITING MODULE
SPEECH MODULE
Cscript code using Microsoft Speech SDK 5.1 speech.js
Speech Grammar math.xml
Handwriting Input using Microsoft Tablet PC SDK
Saves output to file
File testfile.txt
Polls file to look for changes
SPEECH WAIT talktome.lisp
Calls speech-event if there are new speech information arrives
Speech-event talktome.lisp
Inserts into queue with time information
Mouse input
The queue contains information about input type, time, length, and data.
INPUT QUEUE *eventqueue*
Keyboard input
Multimodal Integrator mathinput.lisp
Feeds result to SKEME
Data file
SKEME skeme.lisp mathinput.lisp (and other supporting files)
Data flow
Input
Running Programs

3. Multimodal Input of Mathematical Symbols (Conceptual Model)
Templates for math
Handwriting Input
Voice Input
Keyboard/Mouse Input
Grammar
Initial Data
Ink/Stroke Collector
Speech Recognizer
Events
Separate into Vertical Components
GUI (SKEME)
Recognizer
If confidence poor
EVENT QUEUE
Wait for something on Queue
Wait for more events for X sec.
Data file
Combine the events multimodally
The number of seconds to wait for addition inputs has not yet been determined, but should be in the order of 2 to 10 seconds
Data
Input
Pass event to GUI
Functions