Kingston University Creating and Editing Mathematical Content using Natural Language Commands A Progress Report of the TalkMaths Project Eckhard Pflügel Faculty of Science, Engineering and Computing Kingston University

Overview Motivation and Objective Existing Solutions The Technical Challenges Our Answer: TalkMaths Current Progress Future Steps Conclusion

Introduction Our Motivation: –Not everyone can use a computer with traditional input devices –Mathematical content is notoriously difficult to be accessed with alternative technologies Our Objective: –To carry out research and development that will lead to the creation of a user- friendly tool/system, for accessing mathematical content using speech Kingston University

Speech Recognition Technology Continuous speech recognition has been around for a while Recently, experienced tremendous improvements and attraction in the media (iPhone, Google) Two different architectures: client-based, server-based Why would people want to use it? –Out of commodity? –Out of necessity! Kingston University

Desktop-based Speech Recognition Tools Currently, most reliable (commercial) tool available is Dragon NaturallySpeaking (DNS) An alternative is Windows Speech Recognition (free since Vista, much improved in Windows 7) An open source (free) solution exists – Sphinx Other solutions might appear in the near future Kingston University

Dragon NaturallySpeaking Popular commercial speech recognition product Excellent continuous speech recognition rates Effectively allows for hands-free creation and editing of text Additional features: –Commands for typical computer-based tasks –Supports creation of (application-specific or global) macros Kingston University

Dragon NaturallySpeaking – Drawbacks No support for mathematics (all symbols need to be spelled out) Does not integrate in any mathematical editor Cannot translate speech input into specific mathematical markup (MathML, LaTeX) Development of more sophisticated macros needs (expensive) SDK Anecdotal evidence of mediocre user support Kingston University

Standards for Spoken Mathematics Motivated by different contexts: –Dictating to other human beings –Input for computer systems (parsing) –Audio output for text-to-speech systems Probably first documented source: Changs booklet [1] Fateman [2,3] gives fairly detailed rules for spoken mathematics Other approach: [4] Raman [5] is motivated by TTS Our contribution, based on [2,3]: Wigmore [6] Kingston University

Existing Tools and Systems MathTalk –Collection of Dragon NaturallySpeaking macros –Only usable as input for specific computational maths interface –Commercial product (expensive) –Limited scope Maths Speak & Write –Uses Windows speech technology –Desktop application with GUI –Supports multi-modal input –Research Project –Mainly experimental relevance Kingston University

Related Tool Speed –Goal: programming by voice –Speech-plugin for Eclipse IDE –Uses Java interface for DNS –Speech input with keyboard and mouse –Exports spoken version of Java –This includes navigation –Also research project Overall conclusion: none of the existing tools are fit for purpose. Kingston University

Technical Challenges Research aspect: Need tools and techniques from natural language processing, compiler construction and HCI Standards for spoken maths –Need flexible and powerful grammar –Have to deal with ambiguity Parsing Algorithms –Difficulties: incomplete or incorrect input Editing Paradigms –Need novel strategies for speech-driven UIs

Kingston University TalkMaths Background Use speech recognition for my own work Frequently needed specialist tasks: enter/modify mathematical equations Could use Equation Editor combined with DNS macros More problematic if using LaTeX Idea: write more sophisticated commands Turned this into research project

Our Answer: TalkMaths Web-based User Interface/System Separate the application from speech- front-end Devise special class of speech commands (speech templates) Insight: spoken mathematics can and should be process similarly to spoken structured content (i.e. markup languages, programming code) Kingston University

TalkMaths UI Kingston University

TalkMaths.org Web UI Kingston University php

TalkMaths.org Web UI – Future Version Kingston University

How can TalkMaths Help with Accessibility of Mathematics? TalkMaths aims at people for whom it is –Difficult to use keyboard/mouse –Difficult to decipher equations on screen TalkMaths can help by –Speaking mathematical input –Equation rendering: arbitrary big font sizes –Voice-activated zooming function –Editing mathematical expressions by voice –Planned features: Playback of formulae Importing/exporting existing documents Maintaining documents on server

Progress Report Are developing speech front-end prototype, using Windows speech recognition Still have issues with the DNS interface About to release new version of parser with more robust error recovery New website design in progress Kingston University

Future Work Extending the range of covered mathematics Allowing for multi-line equations/multiple/embedded expressions Potentially, allows sharing of maths input Improved speech editing (select and say) Higher robustness of code/ease of use/installation Improved documentation

Ensuring Continuation Ph.D. student support (full-time until March 2014, part-time until 2017) Industrial funding? EU funding? Domain/hosting not that expensive Kingston University

Conclusion TalkMaths seems to be a novel application TalkMaths can be very helpful for members of Higher Education with computer access problems The current prototype will improve significantly over the next years Keep up to date:

Acknowledgements This work would have been impossible without my colleagues and students: –James Denholm-Price –Gordon Hunter –(and others) Funded by –EPSRC Doctoral Training Award –MSOR Mini Project grant Kingston University

References 1.Lawrence A. Chang. Handbook for spoken mathematics (Larry's speakeasy). Lawrence Livermore National Laboratory, University of California, USA, R. Fateman. How can we speak math? University of California at Berkeley, R. Fateman. 2-D Display of Incomplete Mathematical Expressions. University of California at Berkeley, Cameron Elliott and Je A. Bilmes. Computer based mathematics using continuous speech recognition. CHI 2007 Workshop on Striking a C[h]ord: Vocal Interaction in Assistive Technologies, Games and More, T. V. Raman. Audio system for technical readings. Springer Verlag, Berlin, A. Wigmore. Speech-Based Creation and Editing of Mathematical Content. Ph.D. Thesis, Kingston University, U.K., Kingston University