MUMIS Franciska de Jong & Thijs Westerveld University of Twente Multimedia Indexing and Searching

OBJECTIVES Automatically indexing of video Data from different media sources (paper, radio, tv) Domain: soccer Digitise + ASR Extract significant events Merge annotations Store final annotations UI for searching

FACTS SHEET Title: MUMIS: Multimedia Indexing and Searching Environment Funding: EU Language Engineering Sector of TAP Duration: 30 months July 2000 – January 2003 Volume: 2.4 M Euro, 385 Person months Languages:Dutch, English, German (Swedish)

Consortium University of Twente (NL) Sheffield University (UK) University of Nijmegen (NL) DFKI LT-Lab (DE) Max Planck Institute for Psycholinguistics (DE) Esteam (SE) VDA (NL)

Offline Processing Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Speech Transcr ASR EN DE Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Free Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text IE Merged Annotated formal text NL Information Extraction Automatic Speech Recognition Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Speech Signals Merging Annotations Formal Text Formal Text Formal Text Anno- tations Merging

DOMAIN MODELLING DATA: text, video, audio Location …... Defender a ’Defender’ is a … Player Annotations Multilingual IE Multilingual Search... Player:… Consequence:… Time :… Location:... Multilingual Lexicons ENTITY EVENT RELATION Time Date PersonScoreObject Defender Official Artifact Stopper Goal Player:… Cause:… Time:…... Player Foul

SPEECH RECOGNITION Large-vocabulary Speaker independent Phoneme-based Hidden Markov models acoustic model language model Emotionally coloured speech Domain language model Match specific vocabularies (player names)

INFORMATION EXTRACTION multilingual formal descriptions closed captions tickers newspapers ASR output (radio/TV comment)

IE DATA Formal text Schoten op doel 4 4 Schoten naast doel 6 7 Overtredingen Gele kaarten 1 1 Rode kaarten 0 1 Hoekschoppen 3 5 Buitenspel 4 1 Ticker 24 Scholes beats Jens Jeremies wonderfully, dragging the ball around and past the Bayern Munich man. He then finds Michael Owen on the right wing, but Owen's cross is poor. TV report Scholes Past Jeremies Owen Newspaper Owen header pushed onto the post Deisler brought the German supporters to their feet with a buccaneering run down the right. Moments later Dietmar Hamann managed the first shot on target but it was straight at David Seaman. Mehmet Scholl should have done better after getting goalside of Phil Neville inside the area from Jens Jeremies’ astute pass but he scuffed his shot.

He then finds Michael Owen on the right wing PASS player1 = Scholes player2 = Owen. He Scholes then finds Michael Owen on the right wing … He then finds VP Michael Owen on the right wing NP but Owen's cross NP 24 Scholes beats Jens Jeremies wonderfully, dragging Scholes beat Jens Jeremies wonderfull, drag NUM Scholes PROP beatVERB 3p sing Jens PROP Jeremies PROP wonderfullADV, PUNCT Scholes beats Jens Jeremies wonderfully, dragging the ball around and past the Bayern Munich man. He then finds Michael Owen on the right wing, but Owen's cross is poor. IE Techniques & resources Tokenisation Lemmatisation POS + morphology Named Entities Shallow parsing Co-reference resolution Template filling 24 time Scholes player beat Jens Jeremies player wonderfull, …

MERGING Fuse annotations and recover from errors and differences: Multiple annotations of the same event (possibly with different attributes, e.g. time). Wrong event descriptions because of information extraction errors. Merging multiple partial annotations, e.g. by solving unsolved references like “star player”. Description logic

ON-LINE TASKS Search for interesting events with formal questions (user interface in many languages) Indicate hits by thumbnails & let user select scene Play scene via the Internet & allow scrolling Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Scorer=Overmars; Cause=Head; Time<=45 PSV - Ajax 1995 Ned - Eng 1998 Ned - Ger 1998 Multilingual Search and Display

SUMMARY Multimedia and multilingual ASR on emotionally coloured speech IE on ASR output Merging different annotations Search archives and play video online ml