1 PERSIVAL a System for Personalized Search and Summarization over Multimedia Information

2 PERSIVAL team members ä Medical Informatics ä James Cimino, Carol Friedman, Steven Johnson ä Medical School – cardiac anesthesiology ä Desmond Jordan ä Computer Science ä Steven Feiner, Luis Gravano, Vasileios Hatzivassiloglou, Kathleen McKeown ä Electrical Engineering ä Shih-Fu Chang ä Center for Research on Information Access, Health Sciences Library ä Judith Klavans, Pat Molholt, Elizabeth LaRue, David Millman ä Cognitive Science ä Andre Kushniruk (York), Vimla Patel (Medical Informatics)

3 Students ä Computer Science ä Eugene Agichtein ä Michel Galley ä Noemie Elhadad ä Panos Ipeirotis ä Medical Informatics ä Michael Charney (programmer) ä Eneida Mendonca ä Lyudmila Shagina (programmer) ä Electrical Engineering ä Shahram Ebadollah ä Min-Yen Kan ä Simon Lok ä Smaranda Muresan ä Sergey Sigelman (programmer) ä Yoon –Ho Seol ä Di Wang

4 Goals ä Personalized access to distributed, multimedia resources ä information access ä information fusion ä information understanding ä Provision of patient-specific information ä interaction within context ä for clinicians, at the point of patient care ä for patients, in terms that can be understood ä online patient record serves as a user model

5 Rounds ä Patient-centric ä Current: Access to clinical data ä Missing: Access to literature that fits patient profile

6 Unique Contributions System focus: querying, search, presentation ä Questions are asked within the context of patient information ä A uniform, personalized view of distributed resources on the internet through querying and browsing ä Concise, patient specific presentation of relevant information through summarization ä Access to textual documents linked with access to multimedia video: library of echocardiogram ä Dynamic layout of heterogeneous information

7 Where are we now? ä Prototypes of each system component ä Local library of journal articles and consumer health sites ä 20 highly ranked journals ä 30,000 articles ä Facilities for distributed online search ä Scenarios for development and testing with three patients ä Initial system integration ä Restricted to a limited set of examples ä Formative evaluation of system components

8 Overall Integrated Demo ä What is the prognosis for atrial fibrillation and myocardial infarction? ä Clinician as user ä On viewing patient discharge summary ä Journal articles: controlled clinical trials ä Re-ranking of search results using patient record ä What is the treatment for endocarditis? ä Patient as user ä On viewing lab results ä Consumer health information

11 User Interface Focus ä Asking questions within context of patient record ä Evidence based medicine to suggest questions ä Selection of relevant information from the patient record ä Demo of Medlee

13 Distributed Search ä Meta-searcher for automated interaction with heterogeneous, distributed sources ä Use of machine learning and query probes to automatically determine topics of distributed sources ä Information extraction from web pages

15 Re-ranking search results ä Re-rank articles which better match the patient record -> more relevant articles ä Use natural language techniques to analyze article and patient records ä Articles with many terms and values matching the patient record score higher

17 Presentation Focus ä Multimedia summarization ä Journal articles, consumer health, video ä Highlight retrieved results to help user in finding relevant information ä Personalize summary for patient ä Define unknown terminology ä Methods for summarizing and search echocardiograms ä Dynamic layout and organization of results ä Explicitly control level of detail

18 Milestones ä Where we said we would be vs. where we are: ä Year 2: skeletal end-to-end system prototype with minimal personalization, interactivity, and limited coverage of structured documents ä Year 3: Extend to full prototype, with increased personalization, interactivity, limited coordination of multimedia, full range of structured documents, and restricted coverage of consumer documents ä Use of evidence-based medicine, machine learning to categorize sources by topic, provision of definitions, thin-client computing to allow PERSIVAL on mobile, hand-held devices ä Year 4: Scale prototype with increased robustness, personalization, coverage to full range of documents and fully integrated multimedia. Coordinate with end-to-end evaluation ä Year 5: Refine components based on Year 4 evaluation. Transition PERSIVAL to deployment in cooperation with Health Sciences Library

19 Plans for next year ä Increase robustness ä Extend question asking to different patient contexts, different question types ä Allow summarization and re-ranking of online articles ä Extend journal summarization to new genres ä Extend layout to dynamically incorporate different types of summary input ä Multimedia integration ä Implement scenarios for integration ä Increase interaction with video summary in layout ä Enhanced multimedia prototype