1. Intelligent Information Retrieval and Presentation with Multimedia Databases Floris Wiesman (IKAT/UM) Stefano Bocconi (CWI) Boban Arsenijevic (ULCL/UL) Yulia Bachvarova (CWI) Nico Roos (IKAT/UM) Lambert Schomaker (AI/RUG)

2. 2 Outline From search to presentation I 2 RP architecture Query processing & presentation generation Natural languange generation Ontology mappings Conclusions

3. 3 From search to presentation Standard (multimedia) IR: result of query is ordered list Question-answering system: result of query is answer Our approach: result of query is multimedia presentation containing the answer

4. 4 I 2 RP architecture Presentation generator Ontology agent Semantic network MM DB 1 MM DB n DB 1 ontology DB n ontology Query processor query answer graph Player multimedia presentation Natural language generator facts text

5. 5

6. 6 Query processing and presentation generation Currently simple ‘closed’ queries Answers are determined from semantic network Rules determine which information to present Rules determine which modalities to use

7. 7 Semantic network: Rembrandt’s world Chiaroscuro Caravaggio Caravaggist Italy Pieter Lastman Bible Mythology RembrandtSaskia Rubens teacher Bol is_founded_by paints inspired_by compared_to belongs studies paints studies uses Bent Birds Jan Lievens teacher works_with operated_in Portraits History Paintings paints Night Watch has_genre Hendrickje is_married has_relation Maria Trip has_genre Prophetess Anna has_genre

8. 8 Metadata Search in databases by metadata, no content-based retrieval Metadata has to be such that: –The system can find what it looks for –The system can assemble the retrieved information in a meaningful way –The presentation really means what was intended: a new context is created

9. 9 The information context Information items do not exist standalone, they have a context A presentation needs to combine the retrieved information items in a new context Information sources, from most structured to less structured: –Multimedia Databases (e.g., ARIA) –Digital Library (e.g, Open Archives Initiative) –The Internet

10. 10 Narrative as a context Example of narrative structure (Greimas): –6 Actants: Subject, Object, Helper, Opponent, Destinateur, Receiver –4 Narrative Units: Contract, Competence, Performance, Sanction Every character in the story plays a role in the narrative (identified by rules) –e.g. Artist biography: roles are main character, family members, teachers, collaborators, students Structuring according to role –e.g. family members are grouped in private life section

11. 11 Multimedia presentation

12. 12 Natural-language generation Starts from semantic level Semantic representation may contain: –Participating concepts –Event structure –Temporal organization –Quantification –Relevant discourse functions Transforms selected meaning to natural language sentence

13. 13 Learning ontology mappings The task: –Establish a mapping of concepts in ontology 2 to concepts in ontology 1 Two steps: –Establish joint attention: which are common instances of the ontologies? –Establish mapping: which operations map the concepts best? (wrt joint attention)

14. 14 Two example ontologies Ontology 1 Object Title = Self portrait as St. Paul Artist = Rembrandt Harmensz. van Rijn Materials = oil on canvas Date = 1661-1662 Ontology 2 Artefact Title = Self portrait as the apostle St. Paul Creator family name = Rijn given name = Rembrandt Harmensz. other name = van Material = oil on canvas Period start = 1661 end = 1662 Mapping can be made by copying, splitting, and merging leaf concepts

15. 15 Establishing joint attention do: –agent 1: sends instance of concept to agent 2 –agent 2: returns instance with highest proportion of words in common until enough common instances found above threshold  result is joint attention set: concepts of agent 1 with instances known to agent 1 & 2

16. 16 Mapping of concepts Mapping of concepts consists of operators: –field n: copy leaf concept –merge s: merge leaf concepts using separator s –split s: split leaf concept at separator s –first: copy first part of split –last: copy last part of split Separator: none, space, colon, semicolon (,TC) Example: artefact.period.end  field object.date, split(-), last

17. 17 Establishing a mapping Search space consists of all possible mappings Value of a mapping: number of correct mappings on joint attention set Mapping with highest value wins Search is guided by proportion of words that instances have in common Search space is reduced by ignoring mappings between leaf concepts with a low proportion of common words

18. 18 Conclusions We have shown approach for IR from multimedia databases using: –Knowledge-based query augmentation –Combination of IR results in a single multimedia presentation –Natural language generation –Automatic ontology mapping Parts are realized as prototypes; to be combined in one system