1. Cooperative Computing & Communication Laboratory Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany Multimodal Dialog Description for Mobile Devices

2. © Siemens AG and Paderborn University 2004 Motivating Example Control and monitoring of electronic devices Winamp Mediaplayer PC User Interface Problem: UI does not scale for other platforms / devices / modalities Motivation MIRS DISL Implemen- tation Summary & Conclusion

3. © Siemens AG and Paderborn University 2004 MIRS - Multimodal Interaction and Rendering System Dialog and Interface Specification Language - DISL Structural model Definition of generic widgets (no specific modality) (inherited from UIML with ideas from Paterno, Plomp...) Control/Dialog model Variables, Rules, Actions, Events (inherited from Dialog Specification Notation – DSN) UI Spec (DISL) S-DISL XSLT HW & User Profile S-DISL Interpreter abstract to concrete mapping Renderer Apps Server (Mobile) Client http tcp Motivation MIRS DISL Implemen- tation Summary & Conclusion

4. © Siemens AG and Paderborn University 2004 Dialog and Interaction Specification Language - DISL Modification of UIML Interface Section structure - widget organization Generic widget (modality independent widget) New generic widget elements style- properties defines properties of generic widgets behavior - user interaction defines behavior for generic widgets extension for advanced state-oriented dialog specification states rules events communication Motivation MIRS DISL Implemen- tation Summary & Conclusion

5. © Siemens AG and Paderborn University 2004 1. Container Elements - variablefield - textfield - genericfield - widgetlist - command - confirmation - genericcommand 2. Interaction Elements b. Misc - variablebox - textbox - genericbox - choice a. Input DISL – Structure Elements Motivation MIRS DISL Implemen- tation Summary & Conclusion

6. © Siemens AG and Paderborn University 2004 DISL – Structure/Style/Behavior... Title This is a Remote Control for Audio Playing Devices Press Button to Start Playing true false... Motivation MIRS DISL Implemen- tation Summary & Conclusion

7. © Siemens AG and Paderborn University 2004 DISL - Behavior …............... Motivation MIRS DISL Implemen- tation Summary & Conclusion

8. © Siemens AG and Paderborn University 2004 Events: TogglePower SetVolume ToggleMute PlayTitle StopTitle TogglePause NextTitle... States: Power{#isOn,#isOff} Volume{#0,#1,…,#100} Mute{#MuteOn,#MuteOff} Playing{#PlayingOn,#PlayingOff} Paused{#PauseOn,#PauseOff}... Dialog Specification Notation (DSN) Basic Concepts Motivation MIRS DISL Implemen- tation Summary & Conclusion

9. © Siemens AG and Paderborn University 2004 Rules: Power 1. #isOn TogglePower  #isOff Power 2.#isOff TogglePower  #isOn Mute 1. #isOn #MuteOn ToggleMute  #MuteOff Mute 2. #isOn #MuteOff ToggleMute  #MuteOn Playing 1. #isOn PlayTitle  #PlayingOn #PauseOff Playing 2. #isOn NextTitle  #PlayingOn #PauseOff Playing 3. #isOn PreviousTitle  #PlayingOn #PauseOff Playing 4. #isOn #PlayingOn StopTitle  #PlayingOff #PauseOff Paused 1. #isOn #PausedOn  #PausedOff #PlayingOn Paused 2. #isOn #PausedOff #PlayingOn  #PausedOn #PlayingOff Dialog Specification Notation (DSN) Basic Concepts (cont‘d) Motivation MIRS DISL Implemen- tation Summary & Conclusion

10. © Siemens AG and Paderborn University 2004 States: Power{#isOn,#isOff} Events: TogglePower DSN to DISL true false true false Motivation MIRS DISL Implemen- tation Summary & Conclusion

11. © Siemens AG and Paderborn University 2004 Rules: Power 1. #isOn TogglePower  #isOff DSN to DISL <call source="http://localhost:8080/mediaplayer/servlet" id="tryoff" synchronized="yes" timeout="5s"> turnoff false 2 conditions to be true 2 actions to execute - http request with 1 parameter - 1 variable assignment Motivation MIRS DISL Implemen- tation Summary & Conclusion

12. © Siemens AG and Paderborn University 2004... DISL – Additional Features Change of Interface at Runtime More Complex Expressions... Periodic Events Content of virtually any element can be modified Motivation MIRS DISL Implemen- tation Summary & Conclusion

13. © Siemens AG and Paderborn University 2004 Current Implementation UI Spec (DISL) S-DISL XSLT/Saxon Processor S-DISL Interpreter J2ME/MIDP1.0 AWT-Renderer (J2ME/ MIDP1.0) PC Tomcat http-server http kXML Parser Siemens M55 Now with Bluetooth support!! Motivation MIRS DISL Implemen- tation Summary & Conclusion

14. © Siemens AG and Paderborn University 2004 Summary & Future Work Summary XML-based specification language for general UIs advanced state-oriented dialog specification by variables, rules, actions, events (basic concepts inherited from Dialog Specification Notation – DSN) definition of generic widgets NOT for a specific modality, e.g., text, graphics, speech (basic concepts inherited from UIML, Paterno, Plomp...) Implementation for Java enabled phones Future Work use shorthand vocabulary Better structuring through object orientation (step from DSN to ODSN) Cooperation with high level modelling techniques (bridging the gap between e.g. task models and rendered UI) Proof of concept with more modalities (Speech and Gesture) Motivation MIRS DISL Implemen- tation Summary & Conclusion