Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 1 Informedia 03/12/97

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 2 Multilingual Informedia: Innovations Robust Indexing and Retrieval –Spanish Speech Recognitiion –Searchable User Annotations –Data Extraction for Further Analysis Multilingual Document Access –English or Spanish Queries –English or Spanish Broadcast Video

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 3 Extending the Informedia Digital Video Library Original Informedia Goal –Full content search and retrieval from digital video, audio and text libraries Technology –Integrated speech, image and language processing for automated library creation (indexing, segmentation, abstraction, summarization)

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 4 Building on the Informedia Infrastructure Video and Audio Segmentation –improved segmentation algorithms –extend to multiple languages Presentation, Reuse and Interoperability –abstractions and video summarization (skims) –“cut and paste” for presentations and reports –Annotations Initially typed, later spoken Incrementally indexed for immediate retrieval

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 5 Multilingual Integration –Spanish News Broadcast –Digitized from PAL to MPEG-1 –Speech Recognition/Alignment by Sphinx-III –Simple Phrase-based Translation –Processed Automatically into the Informedia Digital Video Library

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 6 Multilingual Demo Running prototype demo Demonstration of current technologies

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 7 Title Generation for Informedia News Stories Informedia, a multimedia digital library, stores television broadcast news stories. An extractive summary feature currently locates snippets in news-story transcripts to use as story titles. GOAL: An improved, non-extractive title- generation feature for Informedia.

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 8 KNN-based Topic Detection Build training index with pre-labeled topics –45000 Broadcast News stories With new document: Search for top 10 related stories in training index Lookup topics for related stories Re-weight topics by story relevance (select top 5)

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 9 Basic Idea for better Titles Train a statistical model on a corpus of documents with human-assigned titles. Compare title generation methods: –Extractive Titles –Naïve Bayes, EM, –KNN

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 10 Extractive Summarization MS Word 2000 AutoSummarize Extracts sentences/fragments as summaries Similar performance to TF IDF implementation at CMU Does not use our training corpus

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 11 Naïve Bayes Train a statistical model on a corpus of documents with human-assigned titles. Title need not be a snippet from the document (contrasts with extractive-summarization techniques). Suggested by Witbrock & Mittal, P(w Title |w Doc ) –works better if W title = W Doc

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 12 (K) Nearest Neighbor Index a corpus of documents with human- assigned titles. Find the document in the training corpus closest to the current document Use that title (k=1)

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 13 Evaluation of Title Accuracy Apply to unseen documents, (2 * precision * recall) F1 = _________________ (precision + recall) Precision = Correct/Retrieved Recall = Correct/All Possible Correct Only measured word selection, not order Should try String Edit Distance (DTW), or Maximal Substring

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 14 Multi-Lingual Experiment TV news stories with titles from 1998 Broadcast News CD-ROM tested on 1000 held-out stories evaluated on titles Using SYSTRAN (Babelfish.altavista.com) translated English-French-English Vocabulary overlap was about 70% (need) ???

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 15 Example English-French-English Title: CONTINUING COVERAGE OF O. J. SIMPSON CIVIL TRIAL: MORE PHOTOS OF SIMPSON IN MAGLI SHOES MAY SURFACE Translation: AGAIN THE SHOES OF SIMPSON OF O J BECOME A FOCAL POINT IN SA CIVIL TEST. AND AGAIN A PHOTOGRAPH EAST A CRUCIAL PART OF THE IMAGE. FELDMAN OF CHARLES OF C N N EXPLAINS. THE SOURCES INDICATE C N N THAT THE LAWYERS FOR THE FAMILIES CONTINUING SIMPSON OF O J HAVE NOW ACCESS TO SEVERAL PHOTOGRAPHS ALLEGEDLY LATELY CLEARLY DISCOVERED TO SHOW SIMPSON CARRYING A PAIR OF SHOES OF BRUNO MAGLI OF SWEDEN. AN EXPERT AS REGARDS F B I A TESTIFIED WITH THE CIVIL TEST TO SIMPSON TO THAT SUCH A PAIR A LEFT TO THE COPIES TO SHOE BEHIND TO THE SCENE TO MURDER THE FORMER WIFE TO SIMPSON AND HIS GOLDMAN TO RON TO FRIEND. THE AGENT FOR THE FAMILIES OF VICTIMS A PRESENTED IN THE OBVIOUSNESS A PHOTOGRAPH TAKEN BY THE OAR OF HARRY OF PHOTOGRAPHER BY AND PUBLISHED INSIDE QUOTE THE QUOTATION MARK NATIONALS OF INVESTIGATOR. A TESTIFIED EXPERT A THAT PHOTO A SHOWN SIMPSON CARRYING THE SHOES... Method Name Original Machine Generated TitleMachine Generated Title after Translation NBL simpson civil trial simpson's estate news murder investigation simpson civil search victims NBF continuing coverage simpson civil trial verdict continuing coverage simpson civil trial president KNN O. J. SIMPSON TRIAL RESUME MONDAYDAY BACK COURT JURORS SIMPSON CIVIL TRIAL TF.IDF simpson civil trial photo magli shoes continuing coverage simpson civil trial president

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 16 Multilingual Results

Carnegie Mellon © Copyright 2000 Michael G. Christel and Alexander G. Hauptmann 17 Effect of Word Order