1 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 JOANNEUM RESEARCH a TRADITION of INNOVATION Herwig Rehatschek Institute of Information Systems & Information Management Cross Media Aspects in the Areas of Media Monitoring and Content Production

2 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Contents Cross media aspects  in media production (NM2 project)  in media monitoring (DIRECT-INFO project) Conclusions Outlook

3 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 NM2 (New Media for a New Millennium) - Motivation Traditional distribution channels  have developed their specific compelling media forms  art, science and technology of production, direction and editing in non-linear narrative are poorly formed NM2 investigates on  non-linear, interactive narratives  tools for production and delivery Main Objectives of NM2  proof of concept to produce compelling non-linear narratives  production of compelling media aesthetically pleasing truly engaging (30+ minutes) plausible from a business perspective

4 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Basic Technical Concepts Narrative Objects (NO)  MPEG-7 to describe audiovisual features  OWL to describe semantics of NO Narrative Structure Language  internal representation of a „story world“  basic narrative structures

5 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Media Semantic Mapping Utility (Monsoon) Tabs for objects, concepts and relations Definition of an object by low level features Definition of an object by using other objects

6 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 System Architecture metadataessencestories Database interface Middleware API user info ManagementRealisation EngineProfile Learning Ingestion Description Manual Automated Ontology-based Authoring Using narrative structure language Session Management For high volumes End-user Application Including application preview toolsdelivery middleware JRS

7 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Main Project Results Definition of a Narrative Structure Language  describing multimedia objects in narrative context Bridging semantic gap  MPEG-7 & OWL  linking lower level features to higher level concepts Six NL productions with compelling media Targeted for three end-user devices  Windows XP Media Centre and TV card  Game consoles  Mobile phones

8 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 DIRECT-INFO - Background Media Monitoring Companies Measurement of advertisement expenditure Find out  how much a company has spent for advertising  on a product level  over different media Typical business cases  advertisement information  creativity (new campaigns) (targeted in MediaCampaign)  sponsorship tracking (targeted in DIRECT-INFO)

9 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Sponsorship Tracking Sport teams need sponsors to finance their activities Measure efficiency of sport sponsorship  broadcasting & print media  convince sponsors to invest again  currently done manually Technically very complex task  context information needed  multimodal analysis necessary

10 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Main Technical Innovations Identification of logos within visual content Multimodal event modelling Genre classification Text analysis (pos. / neg. mentions) Combination of Semantic correlation & fusion of multi-modal content analysis to semantically meaningful reports

11 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Technical Workflow (1/2)

12 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Technical Workflow (2/2)

13 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 BrandDetector – Detection of Logos

14 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 MPEG-7 Result Viewer/Editor Player Key frames shots Basic events Module parameters

15 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Conclusions Standardized metadata (MPEG-7, OWL) is a good choice  otherwise a lot of time will be lost with specification and extensions  MPEG-7, OWL already provided the needed structures  one expert for MPEG-7, OWL is necessary Web services not so straightforward as always thought  managed vs. unmanaged code  not all SOAP protocol features supported Do not underestimate integration efforts!  take security issues into account  remote access essential

16 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Outlook: MediaCampaign R&D project, Started: 04/2006 Identification and tracking of new campaigns  in different media (Press, TV, Internet)  utilizing different modalities (audio, text, image, video) Semantic cross-market advertis. interlinking  relate similar advertisements over different media & countries Define Ontology for cross-media modeling of media presence and campaigns  Media Presence and Campaign Ontology (MEPCO)  includes media presence information

17 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Relevant Links NM2 website  DIRECT-INFO website  MediaCampaign website  JOANNEUM RESEARCH MPEG-7 Library  available for free!  full support for parts 3, 4 and 5 of the MPEG-7 standard  Available in June 2006!

18 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 JOANNEUM RESEARCH Institute of Information Systems & Information Management Steyrergasse 17 A-8010 Graz / AUSTRIA Voice: Mobile: Fax: Web: Dr. Herwig Rehatschek

20 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Delivery

21 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 MPEG-7 Automatic Detection Preview (UI)

22 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 System Architecture

23 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Genre Classification Fully automatic classification of  commercials  news  studio/speaker  soccer  credits  movie Real-time analysis Semantic block classification  ~90% precision  ~90% recall

24 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Multimodal Event Modelling Event detection in audio and video Integration of basic events  visual shots  audio segments  overlaid text  visible frontal faces Pseudo-scene/story segmentation Classified segments s1s2c1s2a2+s2s3c2c3s4u s5

25 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Text Analysis Detect positive & negative mentions of  brand names  teams SCHUG text analysis engine  “Shallow and Chunk-based Unification Grammar”  robust chunk parser for linguistic analysis Topic detection Sources  transcripts and text captions of TV audio  newspaper texts  supported languages: English, Italian

26 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Semantic Data Fusion Combing data from various analysis subsystems Based on time-stamped analysis results of all analysis modules Rule-based derivation of new facts  basic appearances  complex appearances Two main user interfaces  consolidation of appearances  set-up application for delivery system

27 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Semantic Data Fusion Display of Results

28 20/06/2015 ISO 9001 certified Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Delivery system for Customers