WP5.4/3.1/4.2/5.5 meeting 29th of November 2007, DFKI

Agenda 9:00 - 9:15 welcome, intro and agenda 9: :00 Jan Nemrava: current work in WP5.4/3.1 – video analysis, extended tickers coverage 10: :30 Nikos Simou: FiRE reasoner 10: :00 discussion on connection WP4.2 (FiRE reasoner) with cross-wp work on 5.4/3.1 11: :30 coffee break 11: :00 wp5.5 - IPTC manipulator, KAT integration, discussion about WP5.4 demo 13: :00 lunch

Current work in WP5.4/WP3.1 Textual Sources Video Analysis OCR Reasoning (FiRE) KAMC paper

WP5.4 semantically annotate complementary resources extract semantically organized ‘cross-media features’ from the aligned text and video data sets use in event type classification of video segments by use of fuzzy reasoning

Textual Resources Structured –Players names, numbers, attendance, place, date etc. Unstructured –3 German and 3 English minute by minute reports for all World Cup matches –SPRouT grammar differs for each language –Ita-Fra events extracted from text

OCR (with TUB and JRS) Time synchronization Additional primary resource Merging 16 frames to recognize moving vs. Static objects in the image Text region detection example

audio-video analysis S

audio-video analysis (DCU) Crowd image detector Speech-Band Audio Activity On-Screen Graphics Tracking Motion activity measure Field Line orientation Close-up

audio-video analysis (DCU) Experiments with machine learning –Linear regression used to classify Midfield shot Endzone shot Other (replays, shot on crowd, etc) –K-Nearest Neighbour to classify Cornerkicks –KL-Miner Part of LispMiner developed at UEP Time Series analysis By considering seconds as isolated instances we loose a lot of information

IPTC manipulator (UEP) Extraction of semantic concepts from unstructured texts using DFKI ontology based information extraction tool

IPTC and extraction problems Inability to insert multiple values into certain fields (in case they are duplicate) - city, country, date SPRouT identifies concepts that are in the text but are not depicted on the picture Which information are really relevant for annotation (information about future, past?) (possibility to use sentence syntax analysis to solve this problem) Domain dependency

ITPC manipulator demo

ITPC manipulator Swiss forward Alexander Frei (R) goes airborne challenged by Ukrainian midfielder Oleg Gusev (L) during the World Cup 2006 round of 16 football game Switzerland vs. Ukraine, 26 June 2006 at Cologne stadium. AFP PHOTO / ODD ANDERSEN

A boy shows a football autographed by French midfielder Zinedine Zidane at the entrace of at Weserbergland Stadium in Hameln, 03 July 2006, where the French national team are training ahead of their next match. France on 01 July defeated Brazil 1-0 and will play Portugal 05 July in Munchen in a 2006 World Cup semifinal showdown. AFP PHOTO / DANIEL GARCIA SCHUG output??

Japan's Keisuke Tsuboi, left, and Singapore's Indra Sahdan Daud battle for the ball during the first half in their Asian zone first round of the World Cup soccer soccer qualifying match at Saitama Stadium in Saitama, near Tokyo, Wednesday, Nov. 17, Japan leads Singapore 1-0 at half-time.

Discussion Demo during the K-Space review (Berlin) Complementary Resources Architecture –KAT implementation –COMM ontology with descriptors to express results of comp. resources analysis. IPTC manipulator –Future plans

WP5.4 demo