Video on the Semantic Web Experiences with Media Streams CWI Amsterdam Joost Geurts Jacco van Ossenbruggen Lynda Hardman UC Berkeley SIMS Marc Davis

2 Talk Overview Video Generation Annotating Video/Media Streams Porting Media Streams Video generation using semantic web technology Conclusion

3 Why video? Video is powerful information source but complicated on syntax and semantics done by mostly domain experts. Search/Retrieve is difficult for humans near to impossible for machines. Media metadata is not DC labels, real need for formal semantics Media support is lacking/under developed on the semantic web

4 Why video generation? Practical experience with semantic web technology from a media perspective Media typically is knowledge intensive Ambitious application to trigger problems

5 What can we generate? Cinema is like a language which has structure which can be manipulated –Establishing shot gives the impression an event is happening at a specific location –Reaction shot gives the impression an actor is reacting to an event. Example estab1.smil estab2.smil

6 Continuity editing in cinema Continuity editing rules ensures consistency between shots. –Mise en scene Indoors - Outdoors Clothing/costume Weather conditions –Cinematography Black/white – Color 180 degree rule (walking direction) Framing Jump cut

7 Continuity editing Works best with generic objects/characters Not so well with recognizable objects/characters Theoretic principles in cinema can be formalized and used to automatically generate video sequences.

8 Trip report scenario Student visits Berkeley informs colleagues about his trip. Scenario uses establishing and reaction shots. scenario_iswc2004.smil

9 Scenario – wrap up

10 Generating an establishing shot User: –defines location –defines character Continuity rules: – angle shot 1 > angle shot 2 – Both shots outdoors Annotation requirements on: –Mise en scene (describing scene) –Cinematography (material, lens foci)

11 Annotating Video in Media Streams (Davis 1995) Retrieve/combine existing material for reuse Content driven annotations Multiple descriptive dimensions Icon based 7000 icons in ontology annotations subClass, partOf, looksLike, occursWith

12 Media Streams Timeline

13 Media Streams Icon Space

14 Porting Media Streams to the Semantic Web (syntax) Media Streams application –5 – 10 year old Lisp using old libraries –Programmers moved away –Source code only partly available –Ontology and annotations are in binary format –Not efficiently scalable Reverse engineering –Important: working legacy application Lessons learned: –Knowledge of the application is required to understand the ontology and the annotations. –Focus on syntax

15 Porting Media Streams to semweb (semantics) Ontology: –subClass relationship mapped to rdfs:subClassOf –partOf, looksLike, occursWith relations mapped to generic rdf:property (future work: OWL) –Modularity for streams in ontology XML telephone book RDF telephone book

16 Human cognition vs. machine cognition. Media Streams concepts were represented by icons with a textual label. –In RDF representation there is only the label –Icons can denote complex actions not representable by a single word –Need for WorldNet to retrieve synonyms (hack!) Just RDF is not enough

17 Porting Media Streams to semweb (semantics)

18 Porting Media Streams to semweb (semantics) Lessons learned: –Annotations are not instances of the ontology but refer to it –Structure embodies semantics –Need for annotation template which describes structure

19 Video generation Need for detailed annotations result in detailed queries Detailed queries give fewer results Need to be able to relax query –Specification (ask: pants, retrieve: jeans) –Generalization (ask: jeans, retrieve: pants) System searches for shots which comply with continuity constraints Query or Rule?

20 Conclusion for porting knowledge to semweb Thoroughly understand the original application domain before porting knowledge to the semantic web First focus on porting knowledge to an accessible format such as XML postponing modeling issues. Annotations are not necessarily instances of an ontology but can refer to it, in which case a annotation template defines the structure of the annotation.

21 Conclusion for applications on the semantic web Need for both, precise queries and, queries which allows for relaxing. Distinction between queries and rules is small. Combining proprietary heterogeneous knowledge sources on the semantic web leads to inconsistencies which have to be dealt with.

22 Take home message Legacy sources are because of their longtime existence valuable resources worthwhile porting to the Semantic Web. Best practices guidelines are needed to facilitate this. Ambitious applications test and give requirements on technology

23 Video on the Semantic Web Experiences with Media Steams

24 Scratch Video generation requires detailed, multi dimension descriptive annotations and is therefore a well suited test case for semantic web technology [[read: some knowledge intensive applications, like video generation are dependent on shared knowledge sources provided by the semanticweb. The semantic web should support these. [[read: Video generation is a *real* application giving insights in practical problems with the semantic web]] [[read: technology should subordinate applications]] [[read: ambitious applications show deficiencies in technology]] … … … … </occurence

25 Content structure vs. Documents structure (a) Different ways of annotating: –Structure based (scene, shot, frame) Context important –Content based ( Character walks from left to right”) Context not important

26 Semantics after Syntax (b)

27 Ontology and Annotation (c)

28 Query requirements (d)

29 Rules (e)