1. Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH - Donna Truran Microsoft - Steven Edwards

2. A Language Model of Health Information >80% of information of interest is language. Patients and clinicians use language for > 90% of information exchange. An EMR should be more like a document than a database record. Data Capture is a language processing problem more than a form filling problem

3. Purpose of Hospital Information Systems Retrieve patient records for clinicians Provide data to answer research questions Provide data to answer Management questions Provide clinical alerts for critical incidents Provide decision support for patient care management plan Provide auditing of patient care

4. Data Analytics The primary purpose of HIS is to provide extensive support for patient care, It is not for the medico legal protection of clinicians interests Data Analytics should be the fundamental objective of a HIS The storage repository has more in common with a Content Management System than a relational database IS. Language should be reduced to a canonical form – SNOMED CT

5. Data Entry - Objectives Mimic the workplace processing as closely as possible Identify text as the primary content Make canonical encoding as automatic as possible Make canonical encoding as hidden from view as necessary Maximise flexibility in data entry modes

6. Technology Strategy Multimodal Interface – Developed on the Tablet PC Handwriting & Drawing Capabilities Sub-vocal microphones for speech input – Designed to closely mimic “real” paper forms Generic Form Generation Able to be localised for individual hospitals – Automatically classify Natural Language Classify free text into SNOMED-CT ontology

7. Top Level Overview Interface Form Generator Augmented Lexicon & Standard Lexicon Token Matcher Clinician Database

8. Token Matching Phase 1 – Currently implemented – Matching based off sequence runs of medical terms – Adjacent words compared against each other – Match with most words used chosen as optimal match – SNOMED-CT Description table used; Multiple descriptions map to the same concept

9. Token Matching Phase 2 – To be implemented as future work – If bad matches are found, words close in spelling may be used to accommodate mistakes in the handwriting or speech recognition – Matching algorithm allows inconsistencies/ missing elements in the input – Uses language knowledge to fill in the gaps

10. Token Matching Phase 3 – Also not yet implemented – Uses sophisticated Natural Language Processing techniques to break sentences into “clumps” – Token Matching is then run on the clumps – Allows the negation of SNOMED terms based off sentence clumps

11. Form Generation Necessary attributes of the form are extracted out into an XML format Form generated “on-the-fly” at program runtime Allows hospitals to have non-technical staff use interface generator software to localize standard forms or create their own Output into standard XML for saving into Database

12. Form Generation Next Phase of Implementation – Form can be loaded pre-filled or seeded with data based off statistically average usage – Allow multiple clinicians (Doctors and Nurses) access to the same form at the same time (from multiple Tablet PCs) to speed up data entry and reduce duplication – Add speech recognition and video capture to the interface

13. Conclusion Project outcomes – User Interface was created which closely mimics actual forms currently used in the workplace – Automatically classifies natural language into a medical ontology Performance issues – Classification runs in acceptable time as a background process – Form Generation runs in pseudo-real time – Time for form generation well inside time required to pick up a real paper form

14. Current Progress Building an ED information system based on this model Using Process diagram collated from 3 month study at Westmead ED Subject of ARC Linkage grant application with Sydney West Area Health Service

15. Questions