The KnowledgeMap Project: Development of a Concept-Based Medical School Curriculum Database Joshua C. Denny, MD Plomarz R. Irani Firas H. Wehbe, MD Jeffrey D. Smithers, MD Anderson Spickard, III, MD, MS

Setting Vanderbilt School of Medicine –104 Medical students in each class –4 local hospitals No electronic repository or course schedule

Goals for KM Provide a set of tools to help improve the curriculum and students’ access to it Accommodate a variety of presentation styles Automate document conversion Provide a secure repository of documents that protects intellectual property

KM Structure Web application –Apache web server, MySQL database –Written in Perl, VisualC++, and Visual Basic Multiple servers All documents mapped to UMLS concepts via KM Concept Identifier

Document Corpus Manually converted preclinical lecture handouts –“legacy” documents New handouts/presentations uploaded by faculty

Pilot Anatomy (Fall) and Cell Biology (Spring) for –4 th year elective later came online

KM Concept Identifier Uses NLP techniques –Abbreviation and acronym extraction –Semantic regularization Score-based –Derivational forms (stenosis  stenotic, lungs  pulmonary) –Document-based disambiguation Word and concept clustering Performs favorably with MetaMap on educational documents (82% Recall, 89% Precision)

Document Processing Document uploaded by lecturer, placed in queue Document Conversion Server KM Concept Identifier Document Conversion Server pulls next document off queue, converts to HTML and Text Text version placed in queue Apache Web server HTML & PDF versions Identified concepts indexed for searching

Search Processing User enters a search query, example: “Wilson’s disease” Search Concept Identifier MySQL database C “Hepatolenticular Degeneration” C found in index of curriculum documents

Content Coverage Query Created to answer questions such as “Where is Women’s Health taught?” –“metaconcepts” Uses relationships defined in the UMLS to expand queries with related child and child- like concepts

Content Coverage Query User enters a “metaconcept” query: “Women’s Health” Search Concept Identifier C “Women’s Health” These concepts found in index of curriculum documents KM finds related UMLS concepts MySQL database

Other features Relevant PubMed searches –Based on the document title and the most frequent MeSH concepts in the document Definition searching –Based on UMLS SRDEF file and MedlinePLUS Course management Lecture calendar –Organized by semester and student year

Analysis Primary data source was Log Files: –All events in Apache –Key events in KM, including: Logins/Logoffs Searches Documents viewed (by browse or by search) PubMed searches Content Coverage queries (available only to Course Directors and Administrators) –Removed all events generated by a developer or researcher Downtime measured by a separate server that logged any time the system (or a component) was unavailable

Calculations Browsed document: any documents accessed via a course home page or via the “browse” function on the toolbar Searched document: any document accessed via a search

Results 3271 searches –84% completed with a concept search –15% definition searches (since 5/03) Total of 526 users logged in 15,885 times and viewed 1,143 documents a total of 32,113 times –All members of the first, second, and third year classes have logged on

Results 1264 active documents (1489 total) –722 uploaded by 28 faculty members –135 documents uploaded by authors –407 legacy documents Total downtime was about 20 hours, including scheduled downtime –No true downtime since 1/03

Start of new academic year

* * ***

Conclusions KM is being adopted by more classes –Predominant class and student use still first year courses Students are using KM more –Student use precedes classes coming online –All students with courses online have used KM –Initial reactions seem positive –Heaviest use by 1 st and 2 nd year students

Future Directions Automation of content coverage queries Expand to more courses and another site More types of media Expansions of search algorithms to include spell checker Support for a PocketPC/Palm-compatible site Student tracking

Acknowledgements Randy Miller, M.D. Michel Décary of Cogilex R & D, Inc Dean’s Office Art Dalley, Ph.D. Cathleen Pettepher, Ph.D. For more information