Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany General Chairs Dietrich Paulus Steffen Staab SAMT 2008 Semantics and Digital Media Technology Koblenz, December 3-5, PC Chairs Lynda Hardman Alex Hauptmann Nadia Magnenat-Thalman

Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany Semantic Web as a Knowledge Infrastructure for Multimedia Steffen Staab

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 3 of 28 Depict ISWeb People! Context is missing! Precision 12/12 Recall 12/19 (FTE)  12/33 (all) 

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 4 of 28 Depict ISWeb People! There is a picture of ISWeb in 2006 in the Wiki!

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 5 of 28 Depict ISWeb People! Some more ISWeb members you find on our Web page!

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 6 of 28 Depict ISWeb People! Outside of our Web Site you may also discover current members…

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 7 of 28 Depict ISWeb People! For others, you need to integrate information from the Web and from Word documents of my mailbox

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 8 of 28 Knowledge Infrastructure A Multimedia objects exist in an environment. This environment (context) can and should be used to find, access and present the objects. Knowledge about Multimedia Objects exists in an environment, too. This knowledge infrastructure can and should be used to find, access and present the objects. Well, known in the multimedia field Little explored until now

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 9 of 28 Structure of the Environment  Knowledge in Portal  Knowledge in File System  Knowledge in Wiki  Knowledge in Outside Service  Knowledge in Mailbox

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 10 of 28 Structure of the Environment 1.Collect or Access 2.Harvest/Query Knowledge 3.Collect & Organize Knowledge 4.Use Knowledge

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 11 of 28 What does the Knowledge Infrastructure need? 1.Ontologies  Multimedia Ontologies  Context Ontologies  Domain Ontologies 2.APIs 3.Querying and reasoning

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 12 of 28 How to structure this knowledge? PersonRegion

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 13 of 28 Scenario The "ISWeb" at Bad Kreuznach (team web page)ISWeb Localize a region (person) Annotate the content (interpretation) Carsten Saathoff, aceMedia Project lead, ISWeb, Semantic Multimedia, KAT Annotation Tool Link to knowledge on the Web :Reg1 foaf:depicts kodemaniak:me kodemaniak:me rdfs:label “Carsten Saathoff" kodemaniak:me rdf:type foaf:Person Reg1 Summer School Semantic Web-05 (video)video

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 14 of 28 Problem The "ISWeb" at Bad Kreuznach (team web page)ISWeb Summer School Semantic Web-05 (video)video Seq1 Seq4 Multimedia objects are complex  Compound information objects, fragment identification Semantic annotation  Subjective interpretation, context dependent Linked data principle  Open to reuse existing knowledge  Multimedia O  Domain O  Context O Reg1

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 15 of 28 Reg1 Example: Fragment Identification with COMM People/isweb-2007.jpg core:image-data Image:Yalta_Conference.jpg loc:spatial-mask- role "^^xsd:string dns:realized-by loc:region- locator-descriptor loc:bounding-box dns:plays dns:played-by dns:defines data:has-rectangle People/isweb-2007.jpg

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 16 of 28 Example: Region Annotation with COMM Reg1 People/isweb-2007.jpg foaf.rdf#me What is important here, is what you don‘t see!

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 17 of 28 No am Stef fen “I apply for this position. Here is my cover letter and CV... ” Bercovici.pdf Mrs. Wer ger Bercovici.pdf “Please prepare papers for his hire, Mrs. Werger“ “Thanks for your application and here is some information about the NeOn project…” Colleagues Another Scenario...

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 18 of 28 … Contextualized Information Metadata from contexts other than file browsing is not available, we cannot  inspect conversations in which a file took part  find out about the web page from which it was downloaded  view the event description for which it was created  see the task it is associated with (as input or output) Bercovici.pdf No am “I apply for this position. Here is my cover letter and CV...” Colleagues Context of multimedia information is lost

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 19 of 28 … Communicating VoIP Stef fen No am Communication Context of multimedia information is lost

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 20 of 28 …classifying information… Classification of multimedia information is lost

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 21 of 28 Descriptions and Situations X-COSIM Ontology Sharing, Arguing, Explaining Filing, Downloading, Modifying, Converting Structuring, Summarizing, User-defined Ontologies Creating, Reusing, Copy & Paste

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 22 of 28 DOLCE for Developers of Clients? Particular Endurant Perdurant Abstract Quality Description Situation // Create realization metadata for bikeImage: // Storage on harddisk, Format jpeg FileSystemRealization bikeImageRealization = new FileSystemRealization( bikeImage, new File(new URI("file:/myPics/bikeImage.jpg")), RealModule.I_jpeg); .addAttachment(bikeImageRealizatio n); // Create some addresses and agents carsten and thomas Address addressCarsten = new koblenz.de"); Agent carsten = new Agent(addressCarsten); Address addressThomas = new koblenz.de"); Agent thomas = new Agent(addressThomas); // Create an information object representing the information given by some image InformationObject bikeImage = new InformationObject("bikeImage"); X-COSIMA SPARQLRDF API RDF Store X-COSIMO File Manager, Browser, Mail Client, Task Manager, Text Processor,... Dublin Core, Calendar, Foaf,... Reference Model, Storage Apps, Domain Models Data Access, Mapping Map between contextualized representations and the reference model X-COSIM Architecture

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 23 of 28 Resulting Meta Data Classification Aspect Realization Aspects Communication Aspect Process Aspects No am Stef fen Noam

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 24 of 28 What does the Knowledge Infrastructure need? 1.Ontologies  Multimedia Ontologies  Context Ontologies  Domain Ontologies 2.APIs 3.Querying and reasoning Jump to end

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 25 of 28 Where are these metadata? Process Aspects No am Stef fen Nasir MMDB Server

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 26 of 28 Federation of Knowledge Nasir

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 27 of 28 :NasirFOAF { :NasirFOAF dc:creator :Nasir. :Nasir :depictIn :ISWebWiki { kodemaniak:me :member :ISWeb. Kodemaniak:me :depictIn :WikiImg1. :ISWeb :depictIn Networked Graphs :ISWebFOAF g:definedBy „CONSTRUCT {?X :depictIn...“. } :ISWebFOAF { :ISWeb foaf:name „Info Syst…“. :ISWeb :member :Nasir. :ISWeb :depictIn Images of team members depict part of the group :ISWeb :depictIn Copy images that depict ISWeb

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 28 of 28 :NasirFOAF { :NasirFOAF dc:creator :Nasir. :Nasir :depictIn :ISWebWiki { kodemaniak:me :member :ISWeb. Kodemaniak:me :depictIn :WikiImg1. :ISWeb :depictIn Meta Knowledge :ISWeb :depictIn :ISWeb :depictIn :ISWebFOAF { :ISWeb foaf:name „Info Syst…“. :ISWeb :member :Nasir. :ISWebFOAFMeta { :ISWebFOAF provenance :NasirFOAF. :ISWebFOAF provenance :ISWebWiki. }

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 29 of 28 Take home message  Knowledge Structures of Multimedia  Knowledge Infrastructure  Knowledge Capture Semantic Web ties them together Some first steps are done:  Standardization of Semantic Weblanguages  RDF carried along with some media: RDF in PDF, RDF-A in Open Document Format Interesting Issues:  Standardization of generic protocols  Management of heterogeneity of data, applications and services  Mixed systems of reasoning and low level extraction  ….

ISWeb - Information Systems & Semantic Web Steffen Staab MMKM 30 of 28 Systems & Literature  COMM Ontology & API:  X-Cosim Ontology & API:  Networked graphs:  Querying for meta knowledge:  Distributed querying: ISWC-2007 K-Cap-2007 WWW-2008a WWW-2008b …to be written…

Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany Thank You!

Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany General Chairs Dietrich Paulus Steffen Staab SAMT 2008 Semantics and Digital Media Technology Koblenz, December 3-5, PC Chairs Lynda Hardman Alex Hauptmann Nadia Magnenat-Thalman