XISL language XISL= eXtensible Interaction Sheet Language or XISL=eXtensible Interaction Scenario Language

2 Characteristics XML-based markup language for web-based MMI Grammar is defined by DTD Interaction between a XML/HTML element and a user  User operation (click, speech input,…)  Action (screen update, speech output,…) Separates XML/HTML content from XML interaction Allows user initiative interaction, system initiative interaction and mixed initiative interaction Enables the use of various type of terminals: mobile phone, PC, digital TV, PDA,… Uses I/O cooperativley: parallel I/O, sequential I/O, alternative I Flexibility to expand I/O modalities

3 Goals of XISL Provide a common language for web- based multimodal interaction Satisfy 3 conditions: 1.Control dialog flow/transition: employed from VoiceXML 2.Synchronize input/output modalities:employed from SMIL 3.Modality-extensibility:offered by XISL

4 Exemple

5 Dialog levels 1)Exchange 2)Dialog 3)Document 4)Application 5)Session

6 Dialog levels 1)Exchange: primary unit of interaction between a user and a system non-prompting description (user operation, system action) prompting description (a prompt, user operation,system action) 2)Dialog: composed of exchanges user is always execution a dialog 3)Document: XISL file composed of some dialogs 4)Application: set of XISL documents contains 0..N leaf document, 1 application root document 5)Session begins when user connects to the XISL execution system ends when requested by the user or the XISL execution system

7 Architecture of XISL execution system

8 XISL execution system 1.Front-end module: user interface terminal (microphone, speaker, screen,…) Depends of application and terminal 2.Dialog manager module: interprets XISL documents manages dialog flow controls I and O Independent of application and terminal 3.Document server module (general web server): holds XISL, XML/HTML documents other documents (XISL stylesheet, speech grammar files,…)

9 A subset of XISL elements

10 Tree structure of XISL elements

11 Exemple

12 Dialog manager sub-modules 1.XISL interpreter 2.Input integrator 3.Action module 4.Document manager

13 Outline of XISL execution system XISL interpreter  recives the XISL document from Document Manager  devides it into s : sends to Input Integrator s: sends to Action Module Input Integrator  extracts s from s and sends them to front-end module  parses the s to prepare for input integration  matches the parsed s with the multimodal inputs sent from front-end module  informs the result to the action module Action module  executes action corresponding to the s from Input Integrator  extracts s from s and s  2 types of actions: terminal-dependent: outputs to a user are sent to the front-end Terminal-independent: executed inside the action module (, )

14 Separation of content from XISL document (1) XISL document contains only interaction scenario Content is held separately into XML/HTML files Advantages  system developers can reuse XML/HTML documents and XISL documents  improves readabability

15 Separation of content from XISL document (2) : output to the user Attributes:  type: output modality (browser, speech, audio,…)  event: output event (navigate, close, play,...) : accepts the events raised on XML/HTML documents Attributes:  type: input modality (speech, pointing, key,…)  event: input event (recognize, click, press,...)  match : link to XML/HTML element  return: return values for flexibility strict attribute values and contents of above tags are not specified by XISL (it should be done by terminal developers)

16 Modalities to control interactions in XISL documents: 1.comb attribute in tag Comb=par -> all exchange are executed in parallel Comb=alt -> one of the exchange is executed alternatively Comb=seq -> all exchange are executed in document order 2. the exchange elements bound by these tags have the same semnification as above

17 Exemple

18 Types of front-end 1.PC terminal On Line Shopping application User input: touch-screen display, keyboard, microphone Outputs: touch-screen(anthropomorphic agent), speeaker

19 Input specification for PC terminal

20 Output specification for PC terminal

21 Exemple

22 2. Mobile phone terminal On Line Shopping application Sequential scenario due to terminal’s limits Inputs: user’s speech, DTMF inputs Output:speech

23 Input specification for phone terminal

24 Output specification for phone terminal

25 Exemple

26 3. PDA terminal On Line Shopping application Inputs: cursor key, tap-pen (software keyboard), speech Output: screen, sound device

27 Input specification for PDA terminal

28 Output specification for PDA terminal

29 Exemple

30 Comparation with other MMI approaches (1) 1.SALT: offers tags for describing speech recognition and TTS included in HTML documents Advantages of XISL: better suited to describe complex MMI using sequential, parallel or alternative combination interaction is explicitly separated from XML/HTML contents

31 Comparation with other MMI approaches (2) 2. XHTML+Voice: add VoiceXML to XHTML pages provides voice interaction as additional modalities to web pages VoiceXML comes with interaction flow: control more complex MMI then SALT Advantage of XISL: more flexibility to add modalities

32 Galatea Interaction Builder – prototyping tool Runs on PC Handels input modalities of speech, mouse, keyboard Handels output modalities of speech(TTS), facial expresion, window display Implementation of input and output can be made without the knowleadge of XISL GUI design for domain-specific prototyping (airline ticket reservation, secretary service,…)

33 Interaction Builder

34 Conclusions XML-based markup language for web- based MMI scenarios XISL is separately described from XML/HTML contents (reusability) enables the use of various type of terminals flexibility to describe user’s input and system’s output