© W. Wahlster, DFKI EASSS nd European Agent Systems Summer School Monday, August German Research Center for Artificial Intelligence, DFKI GmbH Stuhlsatzenhausweg Saarbruecken, Germany phone: ( ) /4162 fax: ( ) WWW: Wolfgang Wahlster Generating Virtual Webpages

© W. Wahlster, DFKI First GenerationSecond GenerationThird Generation Static Web Sites Fossils cast in HTML Interactive Web Sites JavaScripts and Applets Database Access and Template-based Generation Dynamic Web Sites Virtual Webpages Netbots, Information Extraction, Presentation Planners Adaptive Web Sites User Modeling, Machine Learning, Online Layout Three Generations of Web Sites

© W. Wahlster, DFKI A Virtual Web Page (Wahlster 1998) is generated on the fly as a combination of various media objects from multiple web sites or as a transformation of a real web page. looks like a real web page, but is not persistently stored. integrates generated and retrieved material in a coordinated way. can be tailored to a particular user profile and adapted to a particular interaction context. has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components. Virtual Memory, Virtual Relation, Virtual Reality... What is a Virtual Web Page?

© W. Wahlster, DFKI Hotel Guide Yahoo News Server Yahoo Weather Server PAN Travel Agent Andi Car Route Planner Gault Millau Restaurant Guide AiA: Information Integration for Virtual Webpages

© W. Wahlster, DFKI Virtual Webpage Retrieved from 5 Different Servers

© W. Wahlster, DFKI Multi-Domain Problem Specs NETBOT Retrieved Results Information Structures Relations, Lists KR Terms Media Objects Texts, Sounds, Videos Pictures, Maps, Animations Distributed Information Multiple Data Sources The Combination of Retrieved and Generated Media Objects for Virtual Webpages

© W. Wahlster, DFKI Retrieved Results Select & Design Select Canned Media Objects Design New Media Objects Information Structures Relations, Lists KR Terms Graphics, Animation Text, Speech, Mimic Icons, Clip Art Frames, Sounds Reuse & Transform Coordinate Media Objects Transform Media Objects Temporal Synchroni- zation Spatial Layout Clip, Convert, Abstract Zoom, Pan, Transition Effects Media Objects Texts, Sounds, Videos Pictures, Maps, Animations The Combination of Retrieved and Generated Media Objects for Virtual Webpages

© W. Wahlster, DFKI Operational Models of Referential Semantics for Robots and Netbots (Wahlster 1999) RobotNetbot “Screw” Physical Objects Screw 1Screw N... Set of Recognizers Set of Subsumption Relations in an Ontology Set of Subsumption Relations in an Ontology “Departure Time” Set of Subsumption Relations in an Ontology Set of Subsumption Relations in an Ontology WWW Objects DT 1DT N Set of Wrappers...

© W. Wahlster, DFKI System is able to flexibly tailor presentations to the individual user and the current situation. Enhancement of User Interfaces through Personalization An animated character serves as “Alter Ego” of the presentation system. Personalized Presenters at DFKI

© W. Wahlster, DFKI The Role of Ontological Annotations for the Generation and Analysis of Virtual Webpages (Wahlster 1999) Information Extraction Agent Presentation Planner Webpages with Ontological Annotations Webpages with Ontological Annotations Webpages without Ontological Annotations Webpages without Ontological Annotations Virtual Webpage Presentation Agent Persona Information Extraction Agents TriAS With Ontological Annotations in: SHOE, OML, XOL,OIL, DAML and Persona Annotation inPML

© W. Wahlster, DFKI

Towards Plan-based Multimodal Interface Agents Action Planning in Robotics Artificial Intelligence Speech Act Theory Philosophy of Language/Linguistcs Plan-based NL Dialog Systems Computational Linguistics Plan-based Multimodal Interface Agents Artificial Intelligence Graphical Acts Semiotics Gestural Acts and Mimics Semiotics

© W. Wahlster, DFKI Netbot PAN Trip Data Pictures and Graphics Pieces of Text Coordinates for Pointing Gestures Input for Speech Synthesis Icons for Hyperlinks Hotel Agent Map Agent Address Weather Agent Train & Flight Scheduling Agent Major Event Agent Virtual Web Presentation Constraint- based Online Layout Presentation Planner Persona Server Components of virtual Webpages AiA The Generation of Virtual Webpages with PAN and AiA

© W. Wahlster, DFKI Persona as a Personal Travel Consultant

© W. Wahlster, DFKI WML- Browser MS-Agent Controller WML SMIL Agent Script PET- PML PET Persona Player SMIL Player AIA’s presentation planner (Andre) has been extended to accommodate for various target platforms through the introduction of a mark-up language layer Presentation Planner

© W. Wahlster, DFKI Persona Server Behaviors Presentation Gestures Reactive Behaviors Idle-time actions Navigation actions Auditory Characteristics Sound effects, auditory icons Voice: male, female Visual Appearances Hand-drawn Cartoon Bitmaps Generated Bitmaps from 3D-Models Video Bitmaps PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications

© W. Wahlster, DFKI Classification of Persona Gestures (Andre, Baldes, Rist) Talking Posture 1 cautious, hesitant appeal for compliance avoids body-gestures Talking Posture 2 active, attentive self-confident uses body-gestures Gesture Catalogue

© W. Wahlster, DFKI take-position (t 1 t 2 )point-to (t 3 t 4 ) move-to (t 1 t 2 )r-stick-pointing (t 3 t 4 ) High-Level Persona Actions Context-Sensitive Expansion (including Navigation Actions) Decomposition into Uninterruptable Basic Postures r-turn (t 1 t 21 ) r-step (t 21 t 22 ) f-turn (t 22 t 2 ) r-hand-lift (t 3 t 31 ) r-stick-expose (t 31 t 4 ) Bitmaps... Context-Sensitive Decomposition of Persona Actions

© W. Wahlster, DFKI Production Act Presentation Act Introduce Create- Graphics S-Show S-Wait S-PositionElaborate-Parts S-Create- Window S-Depict Label S-PointS-Speak S-Point Qualitative constraints:Create-Graphics meets S-Show,... Metric constraints:1 <= Duration S-Wait <= 1,... Distinction between production and presentation acts (i.e. Persona- or display acts) Explicit representation of qualitative and quantitative constraints Extensions of the Representation Formalism

© W. Wahlster, DFKI Persona Presents an Automatically Designed Business Chart

© W. Wahlster, DFKI Virtual Webpage with Animation Effects Based on a Single GIF Image

© W. Wahlster, DFKI Transition Effects in a Series of Retrieved Pictures

© W. Wahlster, DFKI Objective: Enable non-professional computer users to populate their web pages with lifelike characters PET comes with: a set of characters and basic gestures an easy-to-learn Persona markup language Developer’s PET will include: a character design tool which enables users to build their own characters Technical Realization: Based on XML and Java PET: Persona-Enabling Toolkit (Müller, Neurohr)

© W. Wahlster, DFKI Specification of the character to be used Specification of Persona actions Persona Test Features: –XML-based –easy to learn The Persona Markup Language

© W. Wahlster, DFKI PET-Parser Preprocessing (required for non-standard HTML pages) Identification of Persona Tags PET-Generator Replacement of Persona Tag with Java Applet Determination of parameters for Java Applet Realization of Persona active elements as JavaScript Generation of DHTML layers (transparency of Applet box) Generation of data, e.g. audio files for speech output Generation of scripts Tasks of PET

© W. Wahlster, DFKI URL of Webpage with Persona Tag Persona Engine Behavior Monitor Character Composer Event Handler Persona Test Persona Scripts waitscreen 4 gesture greet 0 0 null gesture laugh 0 0 null... Audio Data Bitmaps PET Application Server PET Parser PET Generator Webpage with Reference to Java Applet Functional View of PET

© W. Wahlster, DFKI Text Input Speech Input Menu Input Direct Manipulation Input Web Persona Triggers actions of the Persona Triggers operations on elements of the webpage Mouse Clicks Mouse Movements The Bidirectional Control Flow on Persona-Enabled Webpages

© W. Wahlster, DFKI

some HTML elements  Active Images An active image starts a persona action when clicked.  Addressable Objects An addressable object is an object which can be addressed and manipulated by Persona via its name and its position. Persona Active Elements (PAE)

© W. Wahlster, DFKI Spatial and Temporal Coordination of Multimedia Presentations Multimedia coordination in previous systems: –no declarative representation of spatial and temporal layout –no synchronization of Persona actions with other dynamic multimedia objects Solution: Use PrePlan for automated generation of SMIL expressions –Synchronized Multimedia Integration Language –officially recommended by W3 consortium

© W. Wahlster, DFKI Approach High-level specification of temporal and spatial constraints within the operators of the Presentation Planner PrePlan Use of the incremental constraint solving toolkit Cassowary –Uniform treatment of spatial and temporal constraints –Transformation of qualitative constraints into metric constraints –Integration of a backtracking mechanism to handle disjunctions

© W. Wahlster, DFKI Processing Steps Decomposition of presentation goals into elementary acts and collection of spatial and temporal constraints Constraint propagation using extended Cassowary constraint solver Representation of spatial and temporal layout as SMIL constructs

© W. Wahlster, DFKI Example of a Plan Operator (define-plan-operator :header (A0 (ShowPresentation ?topic)) :constraints (*and*(BELP (Illustrates ?video ?topic)) (BELP (Video ?video)) (BELP (Sets-to-Music ?audio ?topic)) (BELP (Audio ?audio)) (BELP (Summarizes ?title ?topic))) :inferiors (A1 (SAddSmilCode (?video))) (A2 (SAddSmilCode (?audio))) (A3 (SAddSmilCode (?title))) :temporal ((A1 (d) A3) (2 <= begin A1 - begin A2)) :spatial ((aligntop A1) (centerh A1) (centerh A3) (20 <= top A1 - bottom A3 <= 20))

© W. Wahlster, DFKI Representation of Spatial and Temporal Constraints Spatial constraints –Qualitative constraints (Image1 LeftOf Image2), (Image1 TopAlign Image2) –Quantitative constraints (4 < Top Image1 - Top Image2) Temporal constraints –Qualitative constraints Allen constraints: (meets, before, overlaps,...) –Quantitative constraints Linear inequalities: (Begin Audio1 - End Audio2 < 3), (2 <= Duration Audio1 <= 6)

© W. Wahlster, DFKI Resulting SMIL-Document

© W. Wahlster, DFKI

Synchronization of Persona Actions with other Media Objects (Andre, Kleinbauer) Some Examples: point to an object when it appears in a video comment on a video 3 seconds after it has started Smile when video appears on the screen repeat lip movements until audio stops

© W. Wahlster, DFKI Synchronization of Persona Actions (define-plan-operator :header (A0 (ShowPresentation ?topic)) :constraints (*and* (BELP (Describes ?audio ?topic)) (BELP (Audio ?audio)) (BELP (TalkingGesture ?video)) (BELP (RepeatGesture ?video)) :inferiors (A1 (PresentPictureSequence (?topic))) (A2 (SAddSmileCode (?audio))) (A3 (SAddSmilCode (?video))) :temporal ((A2 (d) A1) (2 <= begin A2 - begin A1) (A2 (e) A3)) :spatial ((aligntop A1) (alignleft A1) (1 <= bottom A0 - bottom A3 <= 1) (1 <= right A0 - right A3 <= 1))

© W. Wahlster, DFKI SMIL Specification for Persona Presentation <animation begin= "2.0s" end="15.1" region="reg471101" src= "talking-gst.rp"/>

© W. Wahlster, DFKI Resulting Timeline Diagram At archeological finds in Schwarzenacker,......

© W. Wahlster, DFKI Using SMIL to Synchronize Persona Actions

© W. Wahlster, DFKI

Information Extraction Agents Information Filtering Information Retrieval Information Integration identify relevant documents wrappers –... – identify and extract relevant pieces of information – transform them into canonical form wrappers operational descriptions of a target concept abstract from concrete occurrence within document robust against modifications wrappers operational descriptions of a target concept abstract from concrete occurrence within document robust against modifications

© W. Wahlster, DFKI The Trainable Information Agents Framework (Bauer, Dengler) Browser Application InfoBroker Info Extraction Trainer planning knowledge user preferences domain ontology Web site annotations User requests training specifications results info requests info info requests info or script PBD dialog preferences/heuristics site info/update site information combination of "classical" problem-solving methods and information agents query planning, optimization, and execution improved dialog guidance

© W. Wahlster, DFKI High Degree of Parallelism of Queries

© W. Wahlster, DFKI Knowledge about a Webpage Shared by User and Agent structural visual/semantic procedural Naive User Learning Annotation Agent common part (usable for communication)

© W. Wahlster, DFKI Example - Ontology Train_Connection [ from =>> Location; to =>> Location; travel_date =>> Date; time =>> Time; depart_time =>> Time; arrive_time =>> Time; cost =>> Price; travel_duration =>> Duration; info_url =>> URL;... ]

© W. Wahlster, DFKI Query Planning - I states: information states –concepts / attributes and instantiations operators: querying schemes –preconditions (´+´) and effects (´-´) to time arrive_time travel_duration from travel_date depart_time cost info_url

© W. Wahlster, DFKI Query Planning - IV City.value = München City.language = German... State CityName1 City value language Language Top String... Ontology + + – – babelfish Operators  opprec  cc  ciSi::: 0 )( 0 opIntS 

© W. Wahlster, DFKI Query Plan Visualization Features –alternative queries –past states –future states –state descriptions –PBD requests –accept / reject PBD request –assessment of plans –expected completion time

© W. Wahlster, DFKI Using the presentation planner to serve mobile users (Rist) accommodate for device-specific display and interaction limitations (e.g. 60*90 pixel displays) The new challenge - develop new designs for presenting information on mobile devices - identify elementary design elements - define new presentation strategies for the automated composition of designs Approach

© W. Wahlster, DFKI Application example: Route Descriptions over the Mobile Phone Planner “runway view” “isometric view” “vertical bar view” a selected path gets translated by the planner into a sequence of WML pages to be displayed on a mobile phone

© W. Wahlster, DFKI

Use of a Life-like Character for Electronic Commerce Digital Assistant Selector

© W. Wahlster, DFKI Simulated Dialogues as a Novel Presentation Technique Presentation teams convey certain rhetorical relationships in a more canonical way –Provide pros and cons The single presenters can serve as indices which help the user to classify information. –Provide information from different points of view, e.g. businessman versus tourist Presentation teams can serve as rhetorical devices that allow for a continuous reinforcement of beliefs –involve pseudo-experts to increase evidence

© W. Wahlster, DFKI Presentation Teams (Andre, Rist, Klesen) I recommend you this SLX limousine.

© W. Wahlster, DFKI Underlying Knowledge Base Representation of domain –FACT attribute car_1 consumption_car_1 Value dimensions for cars adopted from a study of the German car market –safety, economy, comfort, sportiness, prestige, family and environmental friendliness –FACT polarity consumption_car_1 economy negative Difficulty to infer implication of dimension on attribute –FACT difficulty consumption_car_1 economy low

© W. Wahlster, DFKI Example of a Dialogue Strategy Question: How much gas does it consume? Answer: It consumes 8l per 100 km. Negative Response: I’m worrying about the running costs. Dampening Counter: Forget about the costs. Think of the prestige! Header: (dampening_counter ?agent ?prop ?dim) Constraints: (*and* (positive ?agent) (pol ?prop ?other_dim positive)) Inferiors: (Speak ?agent (“Forget about the ” ?dim “!”)) (Speak ?agent (“Think of the ” ?other_dim “!”))

© W. Wahlster, DFKI Plan Generation and Plan Recognition as Dual Processes Plan Generation Given: Result: Initial State Goal State Sequence of actions to reach the goal state from the initial state How can I reach my dialog goal? Plan Recognition Given: Result: Initial State Sequence of observed actions Goal State What is the communication goal of my dialog partner? Operator-based Methods Deductive Methods Graph-based Methods

© W. Wahlster, DFKI The generation of virtual webpages is can be achieved by plan- based internet agents. Ontological annotations are needed not only for information extraction agents but also for presentation agents Realization procedures and wrappers form an important part of the referential semantics of objects on the web Using presentation planning with temporal and spatial constraint processing the low-level media synchronization can be done automatically using SMIL annotations see Conclusion

© W. Wahlster, DFKI Computer Science: Elisabeth André Stephan Baldes Mathias Bauer Dietmar Dengler Martin Klesen Thomas Kleinbauer Alexander Kröner Marcus Meyer Jochen Müller Stephan Neurohr Gaby Paul Thomas Rist Wolfgang Wahlster Graphics Design: Bernhard Kirsch Renato Orsini Peter Rist Cognitive Psychology: Susanne van Mulken The WebPersona and Virtual Webpages Team at DFKI