© W. Wahlster, DFKI Third International Conference on Autonomous Agents Agents 99 Seattle, May German Research Center for Artificial Intelligence, DFKI GmbH Stuhlsatzenhausweg Saarbruecken, Germany phone: ( ) /4162 fax: ( ) WWW: Wolfgang Wahlster Personalized Interface Agents for Virtual Webpages

© W. Wahlster, DFKI What is a Virtual Webpage? What are Personalized Interface Agents? Using Life-Like Characters for the Personalization of User Interfaces The Combination of Retrieved and Generated Media Objects for the Generation of Virtual Webpages PET: A Persona-Enabling Toolkit Commercial Applications in Advanced WebCommerce Psychological Effects of Personas Our Research Agenda for Agent-based User Interfaces Multiple Interface Agents Conclusion Outline

© W. Wahlster, DFKI First GenerationSecond GenerationThird Generation Static Web Sites Fossils cast in HTML Interactive Web Sites JavaScripts and Applets Database Access and Template-based Generation Dynamic Web Sites Virtual Web Sites Netbots, Information Extraction, Presentation Planners Adaptive Web Sites User Modeling, Machine Learning, Online Layout Three Generations of Web Sites

© W. Wahlster, DFKI A Virtual Web Page is generated on the fly as a combination of various media objects from multiple web sites or as a transformation of a real web page. looks like a real web page, but is not persistently stored. integrates generated and retrieved material in a coordinated way. can be tailored to a particular user profile and adapted to a particular interaction context. has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components. Virtual Memory, Virtual Relation, Virtual Reality... What is a Virtual Web Page?

© W. Wahlster, DFKI Hotel Guide Yahoo News Server Yahoo Weather Server PAN Travel Agent Andi Car Route Planner Gault Millau Restaurant Guide AiA: Information Integration for Virtual Webpages

© W. Wahlster, DFKI Virtual Webpage Retrieved from 5 Different Servers

© W. Wahlster, DFKI Virtual Webpage Augmented by Persona

© W. Wahlster, DFKI respond immediately to interruptions criticism and clarification questions direct manipulation INTERFACE AGENTS RE- ACTIVE appear as life-like characters plan interactive behavior autonomously can initiate interaction PROACTIVE anticipate the user's needs adopt the user's goals provide unsolicited comments INTER- ACTIVE understand the user‘s requests answer clarification questions allow mixed initiative dialogs What are Agent- based User Interfaces?

© W. Wahlster, DFKI Consumer Provider sells Information Goods Services buys Information Goods Services Web Sites Knowledge about: Usage Patterns User Models Consumer Profiles Netbot Intelligent Parallel Retrieval Information Extraction and Summarization Personalized Presentation Matchmaking Teleshopping Assistance Telemarketing Assistance Translation Services Data Mining Services Intelligent Web Services

© W. Wahlster, DFKI System is able to flexibly tailor presentations to the individual user and the current situation. Enhancement of User Interfaces through Personalization An animated character serves as “Alter Ego” of the presentation system. Personalized Presenters at DFKI

© W. Wahlster, DFKI Persona as a Personal Travel Consultant

© W. Wahlster, DFKI The Personal Picture Finder

© W. Wahlster, DFKI Portrait Photo Netbot: Personal Picture Finder Parallel Meta- Search of Webpages for Parallel Search in Picture Archives Home Pages Extraction of Images Filtering of Logos, Graphics,... WWW Knowledge Sources Applications: Journalism, Contact Preparation, Tracing Criminals A Netbot for Portrait Photos: The Personal Picture Finder

© W. Wahlster, DFKI Personalization = adaption of system behavior according to a user model Personalization = the “agent” appears as animated character which presents, explains and comments an offer, and to which the user can talk to. Personalized Package Presentation by an Animated Agent

© W. Wahlster, DFKI Video Character Presents the Interior of a Boeing 757

© W. Wahlster, DFKI Reactive Behavior of the Persona Agent

© W. Wahlster, DFKI Persona Server Behaviors Presentation Gestures Reactive Behaviors Idle-time actions Navigation actions Auditory Characteristics Sound effects, auditory icons Voice: male, female Visual Appearances Hand-drawn Cartoon Bitmaps Generated Bitmaps from 3D-Models Video Bitmaps PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications

© W. Wahlster, DFKI More than 200 cartoon frames were drawn by a professional artist. A real persona was filmed with a video-camera and the pictures were digitized with a frame grabber. The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons

© W. Wahlster, DFKI The Persona Editor

© W. Wahlster, DFKI take-position (t 1 t 2 )point-to (t 3 t 4 ) move-to (t 1 t 2 )r-stick-pointing (t 3 t 4 ) High-Level Persona Actions Context-Sensitive Expansion (including Navigation Actions) Decomposition into Uninterruptable Basic Postures r-turn (t 1 t 21 ) r-step (t 21 t 22 ) f-turn (t 22 t 2 ) r-hand-lift (t 3 t 31 ) r-stick-expose (t 31 t 4 ) Bitmaps... Context-Sensitive Decomposition of Persona Actions

© W. Wahlster, DFKI Multimedia Generation Text Generator Graphics Generator Gesture Generator Animation Generator Presentation Planner (PREPLAN) KR & Reasoning (RAT) Production Acts Generated Material Presentation Acts Signals + Events Multimedia Server Layout Manager Persona Server Music Generator PPP System Architecture

© W. Wahlster, DFKI Production Act Presentation Act Introduce Create- Graphics S-Show S-Wait S-PositionElaborate-Parts S-Create- Window S-Depict Label S-PointS-Speak S-Point Qualitative constraints:Create-Graphics meets S-Show,... Metric constraints:1 <= Duration S-Wait <= 1,... Distinction between production and presentation acts (i.e. Persona- or display acts) Explicit representation of qualitative and quantitative constraints Extensions of the Representation Formalism

© W. Wahlster, DFKI (define-plan-operator :HEADER (A0 (INTRODUCE P A ?object ?window)) :INFERIORS ((A1 (CREATE-GRAPHICS P A ?window ?object)) (A2 (S-SHOW P A ?window ?object)) (A3 (S-POSITION P A ?window)) (A4 (S-WAIT P A)) (A5 (ELABORATE-PARTS P A ?object ?window))) :QUALITATIVE ((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2)) :METRIC ((10 <= DUR A2) (2 <= DUR A4 <= 2)) :START A1 :FINISH A2) (cf. [André/Rist 97]) Presentation Strategies in PPP contain qualitative and metric constraints

© W. Wahlster, DFKI Multi-Domain Problem Specs NETBOT Retrieved Results Information Structures Relations, Lists KR Terms Media Objects Texts, Sounds, Videos Pictures, Maps, Animations Distributed Information Multiple Data Sources The Combination of Retrieved and Generated Media Objects for Virtual Webpages

© W. Wahlster, DFKI Retrieved Results Select & Design Select Canned Media Objects Design New Media Objects Information Structures Relations, Lists KR Terms Graphics, Animation Text, Speech, Mimic Icons, Clip Art Frames, Sounds Reuse & Transform Coordinate Media Objects Transform Media Objects Temporal Synchroni- zation Spatial Layout Clip, Convert, Abstract Zoom, Pan, Transition Effects Media Objects Texts, Sounds, Videos Pictures, Maps, Animations The Combination of Retrieved and Generated Media Objects for Virtual Webpages

© W. Wahlster, DFKI Persona Presents an Automatically Designed Business Chart

© W. Wahlster, DFKI Virtual Webpage with Animation Effects Based on a Single GIF Image

© W. Wahlster, DFKI Transition Effects in a Series of Retrieved Pictures

© W. Wahlster, DFKI Netbot PAN Trip Data Pictures and Graphics Pieces of Text Coordinates for Pointing Gestures Input for Speech Synthesis Icons for Hyperlinks Hotel Agent Map Agent Address Weather Agent Train & Flight Scheduling Agent Major Event Agent Virtual Web Presentation Constraint- based Online Layout Presentation Planner Persona Server Components of virtual Webpages AiA The Generation of Virtual Webpages with PAN and AiA

© W. Wahlster, DFKI

Objective: Enable non-professional computer users to populate their web pages with lifelike characters PET comes with: a set of characters and basic gestures an easy-to-learn Persona markup language Developer’s PET will include: a character design tool which enables users to build their own characters Technical Realization: Based on standard XML, JavaScript, Java and DHTML PET: Persona-Enabling Toolkit

© W. Wahlster, DFKI Specification of the character to be used Specification of Persona actions Persona Test Features: –XML-based –easy to learn The Persona Markup Language

© W. Wahlster, DFKI URL of Webpage with Persona Tag Persona Engine Behavior Monitor Character Composer Event Handler Persona Test Persona Scripts waitscreen 4 gesture greet 0 0 null gesture laugh 0 0 null... Audio Data Bitmaps PET Application Server PET Parser PET Generator Webpage with Reference to Java Applet Functional View of PET

© W. Wahlster, DFKI PET-Parser Preprocessing (required for non-standard HTML pages) Identification of Persona Tags PET-Generator Replacement of Persona Tag with Java Applet Determination of parameters for Java Applet Realization of Persona active elements as JavaScript Generation of DHTML layers (transparency of Applet box) Generation of data, e.g. audio files for speech output Generation of scripts Tasks of PET

© W. Wahlster, DFKI Text Input Speech Input Menu Input Direct Manipulation Input Web Persona Triggers actions of the Persona Triggers operations on elements of the webpage Mouse Clicks Mouse Movements The Bidirectional Control Flow on Persona-Enabled Webpages

© W. Wahlster, DFKI

some HTML elements  Active Images An active image starts a persona action when clicked.  Addressable Objects An addressable object is an object which can be addressed and manipulated by Persona via its name and its position. Persona Active Elements (PAE)

© W. Wahlster, DFKI Use of a Life-like Character for Electronic Commerce

© W. Wahlster, DFKI Use of a Life-like Character for Electronic Commerce

© W. Wahlster, DFKI Experimental evidence for effects of modality on the user's trust (van Mulken, 1999) The system gives recommendations, which turn out to be wrong in some cases. How much does a user trust the system's advice depending on the modality of a presentation? Self-animated Persona, Speech, Gesture, Facial Expression, Pointing Speech, Graphical Highlighting Text, Graphical Highlighting Personified Agents Increase the User's Trust in the System's Presentation

© W. Wahlster, DFKI Result: Persona > Speech > Text Conclusion: If the presentation is more human-like, recommendations are more readily followed For decision support systems tutoring systems recommendation systems virtual sales agents personified interface agents have a clear advantage: They increase the user's trust in the information presented by the system Impact of the modality of a Presentation on the User's Trustfulness

© W. Wahlster, DFKI Plug-InsApplets Add features (character players) to browser Download triggered by user Requires disk space on client Unrestricted access to client Less appropriate for WebCommerce, Guides Agents integrated in 3D environments Appropriate for Entertainment Examples: Extempo's Jennifer James (Hayes-Roth et al. 98) PFMagic's virtual petz Java animation code sent over the net Automatic loading Requires no disk space on client Restricted access to client Appropriate for WebCommerce, Guides Agents integrated in 2D environments Less appropriate for Entertainment Examples: DFKI's Web Persona (Müller et al. 98) ISI's Adele (Johnson et al 98) New in AiA/PAN: Balanced combination of Applets and Servelets Efficient distribution of client-side Java and server-side Java for driving the Interface Agent Sending Interface Agents to Clients: Plug-Ins or Applets?

© W. Wahlster, DFKI MediaInterface European Media Lab Uinv. Of Munich Univ. of Stuttgart Saarbrücken Aachen Dresden Berkeley Stuttgart MunichUniv. of Erlangen Heidelberg Main Contractor Project Management Testbed Software Integration DFKI Saarbrücken The SmartKom Consortium: Project Budget: $ 34 M Project Duration: 4 years D AIMLER C HRYSLER Ulm SmartKom: Intuitive Multimodal Interaction

© W. Wahlster, DFKI Smartcard/ Credit Card for authentication and billing Docking station for PDA/Notebook/ Camcorder high speed and broad bandwidth Internet connectivity High-resolution scanner Loudspeaker Room microphone Face-tracking camera Virtual touchscreen protected against vandalism Multipoint video conferencing SmartKom-Public: A Multimodal Communication Booth

© W. Wahlster, DFKI MOBILE Camera GPS Microphone Loudspeaker Stylus-Activated Sketch Pad Wearable Compute Server Docking Station for Car PC Biosensor for Authentication & Emotional Feedback GSM for Telephone, Fax, Internet Connectivity SmartKom-Mobile: A Handheld Communication Assistant

© W. Wahlster, DFKI SpeechMike Virtual Touchscreen Natural Gesture Recognition SmartKom-Home/Office: A Versatile Agent-based Interface

© W. Wahlster, DFKI Personalized Presentation Agents (limited user interaction) WIP: Personalized Interface Agents (full user interaction) 2 PPP: Multiple Interface Agents (agent-agent and user interaction) 3 Multiple Presentation Agents in one scene (eg. pros and cons) Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper) Multiple Interface Agents (eg. human-computer, human-human interaction) AiA: Our Research Agenda for Agent-based Interfaces (Wahlster, André, Rist, Müller, Graf etc;

© W. Wahlster, DFKI Use of a Life-like Character for Electronic Commerce Digital Assistant Selector

© W. Wahlster, DFKI But, it’s fast! Multiple Decision Support Agents... weighted propositions User-Adaptive Search Planning Spare parts for this car are rather expensive! Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support

© W. Wahlster, DFKI Multiple Presentation Agents Convey different attitudes, beliefs and biases (e.g. proponent vs. opponent) Generate argumentatives dialogs among a group of agents Multiple Interface Agents human and virtual agents in collaborative browsing environments different dialog roles (e.g. moderator, advisor, sales agent) Multiple Role-Taking of One Interface Agent e.g. travel assistant vs. comparison shopper (use of a different outfit of the character to signal a role change) Multiple Interface Agents

© W. Wahlster, DFKI Multi-User Interaction with Multiple interface Agents for Ecommerce on a Single WebTV

© W. Wahlster, DFKI Advanced WebCommerce Advanced WebCommerce Virtual Web Pages One-to-One Marketing One-to-One Marketing Intuitive, Multilingual Access Dialogue with Virtual Sales Agents Shopbots for Automated Comparison Shopping Text Analysis and Generation User Modeling and Language Generation Coordinated Text & Graphics Planning Robust Dialogue Understanding Advanced Speech Synthesis Information Extraction from HTML/XML Documents Machine Translation Multimodal Interfaces Multimedia Presentation Planning Intelligent Interface Technology is a Prerequisite for Advanced WebCommerce

© W. Wahlster, DFKI Intelligent Web Services Personalized Interface Agents User Modeling Planning Natural Language Understanding Knowledge Representation Image Understanding Machine Learning Plan Recognition Information Retrieval Multimodal User Interfaces Research on Personalized Interface Agents brings disparate subfields in the area of intelligent systems together

© W. Wahlster, DFKI ECommerce projects of DFKI have shown that research on personalized interface agents can be transferred to real world applications: Dekra (largest European organization of used car dealers): FairCar as an ECommerce platform with NL access and a comparison shopping agent for used cars DaimlerChrysler: IKP for online user modelling in a one-to-one marketing system for Mercedes cars Otto/Shopping24/Eddie Bauer (largest European mail order company): virtual sales agents for one-to-one marketing of fashion and computer hardware Lufthansa/Condor: direct marketing of charter flights Conclusion

© W. Wahlster, DFKI The generation of virtual webpages with personalized interface agents leads to innovative applications in: Electronic Commerce, Electronic TV Guides (EPG) Telelearning environments, Call Centers and Help Desks Two Research Challenges: Making the Interface agents sensitive to temporary limitations of the user‘s time and working memory capacity (cf. our READY project, Jameson et al., p in IUI99 Proceedings) Making the agents instructible, so that they can learn from the user in a dialog, how to extract information in a new domain (cf. Mathias Bauer, Dietmar Dengler TrIAs: Trainable Information Assistants for Cooperative Problem Solving in Agents'99, on Tuesday) Conclusion

© W. Wahlster, DFKI Computer Science: Elisabeth André Mathias Bauer Dietmar Dengler Alexander Kröner Marcus Meyer Jochen Müller Stephan Neurohr Gaby Paul Thomas Rist Wolfgang Wahlster Graphics Design: Bernhard Kirsch Renato Orsini Peter Rist Cognitive Psychology: Susanne van Mulken The WebPersona and Virtual Webpages Team at DFKI