1 A Patient Care Digital Library Personalized Search and Summarization over Multimedia Information Kathleen McKeown, Shih-Fu Chang, James Cimino, Paul Clayton, Judith Klavans Columbia University

2 Vision ä Personalized access to distributed resources ä information vetting ä information fusion ä information understanding ä Provision of patient-specific information ä at the point of patient care ä in terms that can be understood ä Benefits: awareness of new procedures, increased compliance

3 Features ä Tailor search, presentation, and summarization of online medical literature ä Exploit online patient record as user model ä Patients and health care providers ä Multimodal access and presentation over multimedia resources

4 Scope of Research “exciting,” “innovative,” “broad,” “ambitious,” “complex” ä Computer Science and Medical Informatics ä Links into libraries: CRIA ä Suite of existing component technologies ä Five year vision ä Active partners

5 Continuum ä Current: existing individual components for multiple applications ä summarization of terrorist news, image search, categorization of images, multimedia briefings, PATCIS, patient mammography record extraction, video summarization of news, query translation ä Future: DLI2 enables ä unification and transformation of components ä new level of capability ä integrated in a single application

6 Outline ä Desmond Jordan ä Why this research is needed ä Kathy McKeown ä An example—integration of components ä Scope, organization, management ä Role of partners and new developments ä Jim Cimino ä Clinical information infrastructure ä Kathy McKeown ä Budget

7 Rounds ä Patient-centric ä Current: Access to clinical data ä Missing: Access to literature that fits patient profile

8 Data follows the patient—why not literature?

9 Ongoing Collaboration: Generation of Multimedia Briefings Voice:.... Ms. Jones is an 80 year old, diabetic, hypertensive, female patient of Dr. Smith undergoing CABG....

10 User Study Group ä Speech, text, and graphics output were useful

11 Multimedia Video ä Physicians: residents learn by comparison ä Patients: ä Video helps them understand their diagnosis ä They like to investigate prognosis Steve Feiner: If I recall correctly, I found the image at the bottom in a we page for patients with CHF. May have something to do with fluid buildup. Steve Feiner: If I recall correctly, I found the image at the bottom in a we page for patients with CHF. May have something to do with fluid buildup.

Why should I prescribe amiodarone? Query History WebSeek Lexical Navigation Steve Feiner: This slide differs from the previous one in that it has a real graphical history, although not in the medical domain. It may be better, since it means that all of the material on the slide will be real. BTW, this is a “comment”. You can either delete it or make it invisible with the “View” menu “comments” command Steve Feiner: This slide differs from the previous one in that it has a real graphical history, although not in the medical domain. It may be better, since it means that all of the material on the slide will be real. BTW, this is a “comment”. You can either delete it or make it invisible with the “View” menu “comments” command

14 Key Issues: User Interface ä Query augmentation using the patient record ä Congestive Heart Failure (CHF), male, Base Heart Rate (BHR) > 90, history of ischemia ä Lucent: speech, IBM: query refinement, interface for image/video search

b Treatme CHF Amiodarone Documents Echocardiogram Echo 7/15/98 Echo 8/15/98 Echo 9/15/98 Echo 10/15/98 Search Augmented query Patient heart sounds Amiodarone CHF male BHR > 90 NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-en the normal mechanism for project reporting to NS Antiarrhythmic Drugs Database of Videos Distributed Search profile

16 Search and Retrieval over Distributed Collections ä Database selection using automatically generated profiles of distributed sources ä Quality evaluation of resource (e.g., NLM points to consumer site on cardiology) ä Concept-based retrieval over full text (GE) ä Search over images using semantic audiovisual templates for event detection

Journal Articles Categorization & Filtering Categorization & Filtering Methods: All patients had advanced CHF with... Amiodarone CHF male BHR > 90 Diagnosis Article NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Amiodarone Documents NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Amiodarone Prognosis Article Prognosis Article

18 Categorization and Filtering ä Categorize document by genre, intended audience, level of complexity, and readability ä Matching documents to patient records

Target Summaries Prognosis articles: ä amiodarone patient, BHR > 90 A significant reduction in sudden death was observed in patients with a BHR > 90 beats/min treated with amiodarone ä pacemaker patient, BHR  90 In patients with a BHR  90 beats/min, amiodarone-treated patients had a mortality rate similar to those not treated with amiodarone. One article found that pacing fails to improve hemodynamic function, while another showed a significantly better quality of life.

21 Presentation and Summarization ä Text summarization ä Extract patient specific information ä Merge repeated and contradictory information ä Select wording the user understands ä Video summarization ä Create hierarchical structure over video ä Detect and highlight differences and commonalities between videos of related cases ä Exploit association between textual information and video

22 Multimedia Presentation ä Coordinated set of displays ä Visual continuity between displays ä Visualization of documents

PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown K. McKeown

J. Klavans PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

L. Gravano S. Johnson J. Cimino S.-F. Chang PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

S. Johnson S. Feiner S.-F. Chang PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

K. McKeown J. Klavans V. Hatzivassiloglou PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

J. Cimino D. Jordan C. Friedman S. Johnson PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

J. Cimino D. Jordan V. Patel PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

P. Molholt S. Jacobson J. Klavans E. La Rue PERSIVAL Management Structure Industry Partners Klavans Clinical Infrastructure Cimino Content and Library Team Klavans Evaluation Patel Project Leader McKeown Technology Search Gravano User Interface Johnson Summarization McKeown

32 Role of Technology Partners ä Lucent: Spoken Language Dialogue Systems (speaker independent) ä IBM: Query refinement, evaluation, text summarization, interface for image/video search ä GE: Concept based retrieval

33 New Partners and Changes ä Siemens Corporate Research: grant for video summarization integrating text/image, plus researcher time ä $68,680/year ä AT&T: grant to build our networking infrastructure ä $50,000 ä Welch Allyn: information and equipment, Medical Director’s time, cash ä $35,000 ä GE Medical Systems: commitment to commercialization of results

34 Partners Steve Feiner: IBM is more than just a software company; ditto Lucent, GE, etc. Why not just group companies together? Steve Feiner: IBM is more than just a software company; ditto Lucent, GE, etc. Why not just group companies together?

35 Partners

36 Clinical Information Systems ä Clinical repository of coded and text data ä Medical Entities Dictionary server ä Web-based access for clinicians (WebCIS) ä Web-based access for patient (PatCIS) ä Infrastructure handles security, application tracking, and evaluation ä Architecture handles integration of components

37 Input for PERSIVAL ä Coded data ä Natural language processing ä Medical Entities Dictionary ä UMLS

42 Web/PatCIS and Digital Library WebCIS/ PatCIS Infobutton Information Resource CISMED

43 Web/PatCIS and Digital Library WebCIS/ PatCIS Infobutton Information Resource Query Interface PERSIVAL CISMED

44 Users ä Clinical users ä Columbia-Presbyterian Medical Center ä New York Hospital ä 37 affiliates ä Patients ä Heart failure ä Home diabetes telemonitoring ä High risk for heart attack ä Cancer patients

45 Evaluation and User-Centered Design ä Usability engineering ä in situ and lab video recordings ä Evaluation of system components: cognitive and quantitative ä Subjective assessment of search plus recall, precision ä Survivability ä end-to-end evaluation, reduction of manual intervention, training, university support of applications

46 Summary ä Tailor search, presentation, and summarization of online medical literature ä Exploit online patient record as user model ä Patients and health care providers ä Multimodal access and presentation over multimedia resources