1. Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T +43.662.2288-200 | F +43.662.2288-222 | info@salzburgresearch.at | www.salzburgresearch.at Towards a Semantic Turn in Rich-Media Analysis 11th International Conference on Electronic Publishing Vienna, Austria - June 15th, 2007 Georg Güntner, Tobias Bürger {georg.guentner,tobias.buerger} @salzburgresearch.at Version 1.0 (June 14th, 2007) © Salzburg Research, 2007

2. 15.06.20072 © Salzburg Research, 2007 Overview |Motivation for Leveraging Media Semantics |Knowledge Models and Media Semantics |Case study 1: Automatic Semantic Analysis Smart Content Factory |Case study 2: Semi-automatic Semantic Analysis LIVE – Live staging of media events |Discussion of Approaches |Conclusions and Recommendations

3. 15.06.20073 © Salzburg Research, 2007 Motivation for Leveraging Media Semantics |The market perspective: | Corporate, community based & personal audiovisual collections (archives, libraries): spanning from YouTube to the big “National Libraries” | Media producers: broadcasters, film producers, game producers | Learning applications, industrial applications | „Embedded publishers“ | Media analysts |The „prosumer crisis“ as a driving factor: | Consumers become producers | Content production gets easier: The amount of manageable assets explodes. | Professional content production is expensive: re-use is mandatory | Solutions of the prosumer crisis depend heavily on improvements of the search and retrieval process

4. 15.06.20074 © Salzburg Research, 2007 Musical forms (z.B. musical genres and forms) Musical facts (Instances: e.g. works, composers, titles) Musical Terminology (e.g. synonyms, classification) Geographic names (places, regions, districts, countries, areas) Domain Ontologies – an Example

5. 15.06.20075 © Salzburg Research, 2007 The Dynamont 3D-Matrix Model Scope Level of Expressiveness Model Acceptance ● which parts of semantics are modeled? ● how is the perspective of the ontology onto the knowledge of the users? ● what kind of semantics is used? ● what kind of semantics are required to fulfill requirements? ● what are the user communities using the ontology? ● which communities accept the ontology? DynamOnt Austrian research project in FIT-IT Semantic Systems (2005 – 2007)

6. 15.06.20076 © Salzburg Research, 2007 Knowledge Models and Media Semantics |Glossary, controlled vocabulary | Finite number of terms, uniquely defined |Taxonomy | Controlled vocabulary, hierarchically structured |Thesaurus | Taxonomy, enhanced by associations, e.g. “A is_related_to B” |Database schema | Typed associations, schema based on a data model |Formal Ontology | A (meta) model, describing possible and true predicates which are assured by axioms © Alistar J. Miles. SKOS Core Guidelines for Migration. http://www.w3.org/

7. 15.06.20077 © Salzburg Research, 2007 Approaches to Bridge the “Semantic Gap” |Construction of meaning | Is an act of interpretation based on pre-existing knowledge (the “context”) | Can usually not be extracted by low-level feature analysis  Semantic gap Solutions:  Creation of training data sets  Analysis of usage context  Automatic, semi-automatic and manual annotation  Combinations of above annotation approaches approaches  Social Web approaches

8. 15.06.20078 © Salzburg Research, 2007 Case Study 1 Smart Content Factory |Vision: | Development of a prototype of a knowledge-based audiovisual archive and navigation system for TV, radio and online broadcasts | Increasing the utilization of audiovisual content repositories |Duration: 10/2003 - 09/2006 |Volume: 735.000 € |Efforts: 80 PM |Partners: | Salzburg Research (co-ordinator) | ORF | X-Art ProDivision | Joanneum Research http://scf.salzburgresearch.at/

9. 15.06.20079 © Salzburg Research, 2007 Smart Tracks Time Metadata Tracks: Watch Listen Read Face ID Speaker ID User Annotation Speech to Text Timecode Keyframes Closed Caption Lynne Russell Hillary Clinton “Clinton spoke with reporters” “Arrived in New York” “Older citizens who have” Improving Medicare Speaking at rally Clinton running for senate Hillary Clinton Lynne Russell 00:08:40:0000:08:26:1200:08:34:29 Encoded Video >>Hillary Clinton Spoke with reporters when she arrived in New York about her proposal for older citizens... Audio Classification SpeechApplauseSpeech On-Screen Text Special Report Medicare Issues © Virage

10. 15.06.200710 © Salzburg Research, 2007 Location Based Navigation (using Google Maps™)

11. 15.06.200711 © Salzburg Research, 2007 Browsing by Category

12. 15.06.200712 © Salzburg Research, 2007 Case Study 2 Live Staging of Media Events |Vision: The central idea of Live is to create | Novel Content Production Methods | Tools for Interactive Digital Broadcasters | New ITV video formats and services to stage Live Media Events such as the 2008 Olympic Games. |Duration: 01/2006 - 09/2009 |Volume: 11.3 mio. € |Efforts: 1165 PM (~100 PY) |Partners: | Fraunhofer IAIS, Cologne (DE) | Academy of Media Arts, Cologne (DE) | ATOS Origin, Madrid (ES) | Austrian Broadcasting Corporation/ORF (AT) | Pixelpark (DE) | Salzburg Research (AT) | University of Applied Sciences, Cologne (DE) | University of Bradford (UK) | University of Ljubljana (SLO)

13. 15.06.200713 © Salzburg Research, 2007 Modelling the LIVE Staging Domain

14. 15.06.200714 © Salzburg Research, 2007 LIVE System Components Related to Media Semantics

15. 15.06.200715 © Salzburg Research, 2007 Knowledge (ontologies) Media Asset (metadata) Essence (videoclip) Intelligent Content Models for Leveraging Media Semantics (1) Resource Level (2) Meta Level (3) Subject Matter Level Model

16. 15.06.200716 © Salzburg Research, 2007 Discussion of Approaches (1/2) |Enrichment of multimedia data | Analysis of usage context | Analysis of different modalities | Human annotation | Use of Web2.0 approaches |Combination of low-level feature extraction with background knowledge | Rich News combines text extraction with Google searches | Google extracts topics from the subtitle channel and links to related web pages | In intranets: combination with production metadata

17. 15.06.200717 © Salzburg Research, 2007 Discussion of Approaches (2/2) |Using different modalities | MediaMill: e.g. when visual content is not reflected in closed caption or in the audio track | MISTRAL: “Measurable intelligent and secure semantic extraction and retrieval of multimedia data“; Austrian FIT-IT project | Combination of media asset management approaches with document management technologies | Virage, Autonomy, Convera, etc. | Google, Yahoo, etc. |Combination of media asset management approaches with social software approaches | Web 2.0 platforms (some of them recently acquired by the big players in the media community)

18. 15.06.200718 © Salzburg Research, 2007 A Recipe for the Semantic Turn in Rich-Media Analysis |Use existing content (URIs) |Define an appropriate content model | Is the content invariant to the semantic approach? Must the structures change? |Integrate existing information sources | Production metadata (speakers, editing), archival data, playout systems, EPGs | Consider creating links to other sources |Test metadata extraction tools |Consider metadata annotation | Is social tagging an alternative? |Define an appropriate domain knowledge model | Scope, expressiveness, acceptance |Implement the semantic indexing process |Integrate the system in the production / archiving workflow

19. 15.06.200719 © Salzburg Research, 2007 Conclusions & Recommendations |Define the scope of your application! |Create persistent URIs (J. Hendler) |Include reliable information sources! |Leave data in place – rather integrate through an RDF store! (J. Hendler, Semantics 2006) |Use intelligent content models support semantic search! |Consider social software approaches where applicable! |Train your staff! |Define the role of the archivist! |„Install“ a knowledge engineer!

20. 15.06.200720 © Salzburg Research, 2007 Thank you for your Attention! |Additional information | Salzburg Research: www.salzburgresearch.at | Salzburg NewMediaLab: www.newmedialab.at | Smart Content Factory: scf.salzburgresearch.at | IST-Project LIVE: www.ist-live.org |Contacts | Güntner Georg T: +43.662.2288.401, M: +43.664.2807149 georg.guentner@salzburgresearch.at Salzburg Research Project manager of Smart Content Factory Scientific and technological coordinator of LIVE Salzburg Research Jakob Haringer Straße 5/III A-5020 Salzburg Austria/Österreich Tel. +43.662.2288.400 Fax: +43.662.2288.222