CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try

Presentation Agenda DWARF 簡介 Introduction UIML 簡介 Requirement Analysis Related Work Design Example Future Work Value to our research

Reference [1]Christian Sandor,Thomas Reicher(2001), “ CUIML: A language for generating multimodal human-computer interfaces ”, Proceedings of the European UIML. [2]Marc Abrams,et al., ” UIML:An Appliance-Independent XML User Interface Language ”,Garvin Innovation Center,VA. [3]Micah Dubinko(2001), ” What Are XForms ”, l. [4]Just van den Broecke(2000), ” Pushlets: Send events from servlets to DHTML client browsers ”, 2000/jw-03-pushlet_p.html. [5]Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas Reicher, Stefan Ri ß, Christian Sandor, Martin Wagner (2001), ” Design of a Component- Based Augmented Reality Framework ”, ISAR 2001

DWARF 簡介

DWARF: A Wearable Computer

Augmented Reality 虛擬物件與現實 世界的結合

DWARF 架構圖 CUIML

Introduction CUIML:Cooperative User Interfaces Markup Language. CUIML is part of DWARF project The DWARF is to develop a framework for applications on wearable computers which consists numerous devices. The interaction with the user should be multimodal in this situation.

Multimodal System Multimodal System supports communication with the user through different modalities such as voice,gesture and typing Modal means “Modality” and “Mode” Modality Communication channel used to convey information. Mode State that determines the way information is interpreted to extract meaning.

Multimodal System(2) In the multimodal HCI,system should support user input and system output by various ways. Examples Input:speech,gesture,keyboard Output:voice(VoiceXML),3D graphics(VRML),2D graphics,Text(HTML). In UIML,we want to describe HCI using an abstract language,so this language should allow us to define the behaviors of various input / output components.

UIML 簡介 Create natural separation of UI design from programming Job. Allow many to one relationship between the two. Transform to native markup language by XSL A measure of the power of UIML is that it can replace hand-coding of Java AWT/SWING user interfaces. ( 和 SALT 有何不同 ?)

Where UIML Fits In Application database Content (XML, audio,…) Device Adaption UIML WMLVoiceXMLXHTMLcHTML

UIML Perspective Machine language Assembly language “High-level” language (C++, Java) Scripting languages Device-dependent markup Device-independent markup

Features of UIML UIML can be rendered to many device-dependent language like VoiceXML,HTML or WML. Web-based Interfaces generated with UIML can send events but can not receive events.( 每次改變時，都要重新送 一次文件 ) UIML only generate views that do not interact with each other.

CUIML Overview CUIML use UIML as a starting point. CUIML is a unified user interface description language to describe multimodal user interface in an abstract way. CUIML suggests a MVC(Model-View-Controller) framework to solve the synchronization problems.

CUIML Overview(2) By XSL transformations,these descriptions are converted into various markup language for various devices. The controller’s internal structure is a DFA(deterministic finite automaton),it map CUIML to various input and output device

Requirements analysis Modeling information at a high level of abstraction Mapping generic information to device-dependent presentations Synchronization of the different channels Controller configuration

Related Work UIML Petri Net S.Riss Work Flow Engine XForms Mozquito

Design of CUIML

Runtime behavior of the components The client request a HCI by sending a HTTP request. Server renders the components and send back to client. The rendering is done by applying XSL transformations.

Initializing the Client Controller Config / Information for Manipulator

Generation of HCI components

The generated Components(1) Controller Synchronize the view components Keep track of current state of HCI Use WFE implementations in prototype

The generated Components(2) View Display markup languages Send event to controller The concept of describing and rendering views is adopted form UIML.

The generated Components(3) Manipulator An enhancement over UIML. Changes to the view is done by manipulator instead of rendering new view. View is accessed by manipulator by the Document Object Model(DOM).( 這代表 View 不能太大 )

The generated Components

Structure of CUIML

Create Multi User Interface with CUIML

Example:Sticky Selected Edit this sticky Edit here 紅色代表目前被選擇的利貼

DFA of the Controller

Define the interface

Define the events

Future Work To develop a CUIML IDE Security concerns of the CUIML component sent to client side Internal structures of Controller using DFA is not sufficient for more complex situations.One possibility will be Petri Nets or even Colored Petri Nets.

The value to our research The idea of synchronize VRML 、 VoiceXML and HTML with MVC Model.(Instead of time tag) To change views at runtime by Manipulator. The flow control language may be substitue by XLANG or Petri Net.

Problems may solved using CUIML Voice /Animation synchronization. Protocol integration. ASR/TTS integration and its performance issues. Virtual user autonomy. The “Voice propagation range” issues.

System Design Prototype(Before)

System Design Prototype(After)

Q & A