1Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML: an Informal Design Tool for User Interface Early Prototyping Adrien Coyette, Jean Vanderdonckt, Quentin Limbourg Université catholique de Louvain (UCL) Louvain School of Management (IAG) Information Systems Unit (ISYS) Belgian Lab. of Computer-Human Interaction (BCHI) {coyette, vanderdonckt,

2Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 What’s the situation of today? Interactive Software evolution: context of use =(U,P,E) time Platform User Environment Language

3Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 What is the situation today? Technological aspects of user interfaces progress significantly faster than –Software engineering aspects It takes time to develop a user interface with a new device, a new interaction technique It takes more time to develop a toolkit It takes even more time to rely on a model-driven approach –Usability engineering aspects New user interfaces are shipped with usability problems because –Little or no experience –No past, no empirical evidence Empirical experiments require a lot of resource if done carefully

4Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 How to address this issue? Capture essence of concepts through models –Separation of concerns, Correlability –Parsability, editability –If possible, human readability Typical models

5Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Typical models

6Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Mappings across levels Limbourg [Limbourg, 2006] Environment T Final user Interface T Concrete user Interface T Task and Domain T Abstract user Interface T T=Target context of use Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use Reification Abstraction Reflexion Translation UsiXML unsupported model UsiXML supported model User S Platform SEnvironment SPlatform T User T

7Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Our goals Objectives –Description of interactive systems Using pre existing domain specific meta-models Used both at design and run time –Capitalization Properties Transformations Interactive system Model of the IS Model of the IS ‘ Interactive system ‘ Checks of properties Transformation Models, instances of Meta- Models described in MOF (even for properties and transformations…)

8Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 UsiXML UsiXML = –USer Interface exTensible Markup Language –XML-compliant specification language for user interfaces suitable for any interface Web Java Windows-based applications Multimodal applications, 3D applications Virtual, mixed reality applications – –Join the UsiXML Consortium by registering on line

9Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 All the models in UsiXML

10Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 The Concrete User Interface in UsiXML Simple widgets (e.g., check box) Containers (e.g., dialog box) Behaviour of Concrete interaction objects Relationships between Concrete interaction objects

11Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 MDE based on UsiXML Model to Model Platform Independent Model (PIM) Platform Specific Model (PSM) Model to Code Source code MDE Components Techniques proposed based on UsiXML Computing Independent Model (CIM) Model to Model UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Graph transformations Graph transformations

12Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 The whole picture of UsiXML according to MDE UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Generative programming Graph transformations Graph transformations Derivation rules IdealXML ReversiXML FlashiXML QtkXML InterpiXML VisualiXML TransformiXML GrafiXML VisiXML SketchiXML FormiXML KnowiXML MethodiXML

13Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan 1.UsiXML 1. Overall presentation 2. SketchiXML 1.Motivations 2.Paper prototyping 3.Computer Assisted Prototyping 4.SketchiXML Development 5.Surveys 6.Conclusion

14Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations user interface determines how easily a user may control underlying functions of a computer program. Thus, a program equipped with powerful functionalities and low quality user interfaces may be under- exploited or misused

15Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations Additionally, the proportion of code dedicated to user interface in application development is considerable: 50% to 70 % from 50% to 70 % of the application code [Myers and Al, 2000]

16Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 How to build a graphical user interface efficiently? Start  all programs  JBuilder  build your UI Motivations

17Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations Even if a “final result” is quickly available, the UI created is likely to require a huge set of expensive modifications… There are no rewards in life for being the first one with the wrong answer. Snyder [Snyder, 1996]

18Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations  As it was the case in software engineering, HCI communities wanted to develop a well- structured method for developing user interfaces.  However, the systematization, and the reproducibility found in software engineering methods cannot be transposed equally in HCI: the development life cycle remains inherently open, ill-defined, and highly iterative Limbourg [Limbourg, 2004]

19Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations  programming approach (1):  direct implementation of a final interface  exploratory approach (2):  quickly collecting user's feedback  specification-based (3):  specify relevant properties of a UI at various levels of abstraction Barthet [Barthet, 1988] Diane Methodology

20Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations  We consider that informal techniques based on low fidelity mock up is the best alternative for the early design phase.  It allows to cope with the fact the such process is eminently open, iterative and incomplete Sumner [Sumner, 1997]

21Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations  Such approach allows to reduce the time needed between the cycle  Permit to involve the end user in the process since few technical background is required  And rises as many usability problems than using high fidelity mock-ups Virzi and Al [Virzi and Al, 1996]

22Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Motivations Virzi and Al [Virzi and Al, 1996] FidelityAppearanceAdvantagesShortcomings Low -Sketchy -Little visual detail - Low development cost - Short production time - Easy communication - Basic drawing skills needed - Is facilitator-driven - Limited for usability tests -Limited support of navigational aspects - Low attractiveness for end users - No code generation Medium - Simple - medium level of detail, close to the appearance of the Final UI - Medium development cost - Average production time - May involve some basic graphical aspects as specified in style guide: labels, icons,… - Limited drawing skills - Understandable for end user - Is Facilitator-driven - Limited for usability tests - Medium support of navigational aspects - No code generation High -Definitive, refined - Look and Feel of final UI - Fully interactive - Serves for usability testing - Supports user-centered design - Serves for prototype validation and contract - Attractive for end users - Code generation - High development cost - High production time - Advanced drawing and specification skills needed - Very inflexible with respect to changing requirements

23Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

24Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping What is it used for?  Early design phase  Get a sense of the user's needs and goals  Quickly prune the tree of possible designs  Remove “locked-in” choices Snyder [Snyder, 1996]

25Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping How to Create a paper prototype ?  Paper  Eraser  Pencils  Scissors  Glue/Tape  Post-it Snyder [Snyder, 1996]

26Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Snyder [Snyder, 1996]

27Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Paper prototypes are as much about:  seeing how the user uses the design DURING  as it is about seeing what ideas (user and designer) come up DURING the design Iteration is key Snyder [Snyder, 1996]

28Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Design Tab How to do it?  Brainstorm, Concept  Design  Test Snyder [Snyder, 1996]

29Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Brainstorm  Come up with all the tasks and goals you think the user will need in the interface  Come up with all the individual functions that will be performed in the interface  Generate a concept of how each goal will be accomplishedDesign  Create paper mockups of a series of screens representing each feature  One screen for each step in each task  Generate a diagram of “ state transitions” from one mockup to another Snyder [Snyder, 1996]

30Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Test  Pick users and have them step through common tasks  As they move through the steps, describe what happens after each action, and show them the next screen  Write down mistakes, misunderstandings, problems, things that workedIterate  Using what you learned, redesign the paper prototype to improve the design  Retest and redesign until all common problems are removed Snyder [Snyder, 1996]

31Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Paper prototyping appears thus to be the a natural answer:  familiar and unconstrained approach  fast to learn and quick to produce  focus on basic structural issues instead of unimportant details  it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,  can be performed collaboratively between designers and end-users

32Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Design Tab But drawbacks also exist…  changes are hard to accomplish  The designer has to redraw the common features that the design retains  lack of interaction between the paper-based design and a user (a designer needs to"play computer" )  Can't emulate complex interactions But...it isn't designed for those purposes: It is designed to solve problems before any money or time is invested in a solution

33Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping Designers almost never use paper prototyping in design projectsWhy?  Doesn't impress?  Too simple and cheap to do anything valuable?  Probably a little of both Snyder [Snyder, 1996]

34Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Paper prototyping  The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations  Such tools would extend the advantages provided by sketching techniques by easily creating, deleting, updating or moving UI elements.  Some research was carried out in order to propose a hybrid approach, combining the best of the hand-sketching and computer assisted interface design, but as we will se trough the state of the art, this marriage presents a set of shortcomings.

35Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

36Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Computer Assisted Prototyping DENIM (James A. Landay, James Lin, Mark W. Newman, Jason I. Hong)

37Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Computer Assisted Prototyping Advantages:  Support for scenario-based design  Several levels of granularity  Good documentation  Toolbox of generic widgets  Mature product, based on experimental testing  Zooming facility from local design (e.g. a web page) to a global design (e.g., a portion of a web site or an entire web site)  Storyboarding facilities based on patterns

38Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Computer Assisted Prototyping Shortcomings:  No shape recognition and interpretation, thus loosing the effort  No code generation  No preview mode  Only dedicated to web sites

39Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Computer Assisted Prototyping Advantages  Performance (speed and accuracy)  Multi-stroke gestures  Recognizes rotated shapes  Computationally light  Open source  Requires standard and freely available libraries  Extensible shape interpretation grammar

40Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Computer Assisted Prototyping Shortcomings:  No scenario editor  Mono-window  Only generates java (and no UI spec)  Limited widget set  Shape interpretation can only take as input a construct made of a maximum of two vectorial shapes  No zoom

41Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

42Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development  Avoidance of Effort loss  Large conceptual coverage  Robust recognition  Language neutrality  Ease of use (naturalness)  Flexible processing  Context independency Requirements elicitation:

43Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development  Context independency  Well defined editing functionalities.  Save and export facilities.  Zooming capabilities  Robust scenario editor  Recognition flexibility … Requirements elicitation:

44Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Multi-Agent System SketchiXML as a Multi-Agent System  Adaptive  Modular  Flexible

45Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Software Agent Implemented with/in software technologies Environment : humans, machines, other software agents, platforms. Agent : Agent : Individual who can act Intelligenceautonomous, pro- active, adaptative with/in its environment  Intelligence Multi-agent system: organization of individuals to achieve particular, possible common goals.

46Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development I* representation of the Virtual mediator : When a shape is provided, the mediator send it to the appropriate agents for recognition and integrate answers

47Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Recognition engine :Cali library  A fast, simple and compact  A fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes) temporal adjacency, Fuzzy Logic and geometric features  Combines temporal adjacency, Fuzzy Logic and geometric features to classify scribbles.  Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap. recognized independently of changes in rotation, size or number of individual strokes  Shapes are recognized independently of changes in rotation, size or number of individual strokes. 97%  The recognizer has a recognition rate of 97% and a fast response

48Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development  Using the cali library: –widgets have to be “constructed” based on geometric primitives (elipse, rectangle, line, …) –each geometric primitive is entered (drawn) separately The new approach: –user can give examples for each widget (training) –complete widget can be entered all at once –distance measure between trained and drawn widgets based on stroke directions – Multi-strokes too (!= Rubine’s algorithm)

49Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Recognition engine :Trainable Gesture Recognizer  Handdrawn input (sketch) is superimposed with a grid  Each sampling point is mapped to its closest grid node  For adjacent nodes, a stroke direction (out of 8 different directions) is taken  A string is generated from direction coding of input  Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances

50Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Distance is shortest sequence of edit commands that transform s to t Simplest set of operations: Copy character from s over to t Delete a character in s (cost 1) Insert a character in t (cost 1) Substitute one character for another (cost 1) Levenstein distance

51Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development shape interpretation The shape interpretation is based on the same mechanism:  When the program starts, the visual grammar is loaded For each widget representation, an agent is created and try to build a widget according to his dedicated pattern Each of these agents register to the mediator, and provide the kind of shape required according to its pattern

52Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Drawing Canvas Shapes Recognition Shape Interpretation Update Beliefs Update UI SketchiXMLprocess

53Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 SketchiXML Development Check box Identification label A variable degree of fidelity Low Medium Final

54Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

55Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys Building the widget catalogue

56Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

57Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

58Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys Recognition rate of the application Overall recognition rate for widget : 94.12% (only) No difference between end user and computer scientist

59Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys Evaluation of the usability of the application: 1.Evaluation of the time needed to draw a simple form with the application:  The time needed decrease strongly as user get used to the application (learning effect)  End user are significantly faster than computer scientist to draw the same user interface 2. Evaluation of the application with a multiple choice questionnaire at the end of the test  Most of the participant enjoyed the test  Most of them found the application easy to use  Most of the participant were satisfied with the results (despite the low recognition rate)

60Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

61Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

62Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

63Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

64Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

65Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Surveys

66Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

67Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Demonstration

68Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Presentation plan  Motivations  Paper prototyping  Computer Assisted Prototyping  SketchiXML Development  Surveys  Demonstration  Conclusion

69Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Conclusion We have introduced a new and innovative sketching tool that generates a user, platform, and environment independent output We have met most of the requirements that were identified as important shortcomings of existing tools we have also conducted a survey on 60 peoples from different activity sectors with different backgrounds, in order to identify how these people would intuitively represent the widgets to be handled by SketchiXML We have conducted a survey on 40 peoples with different background in order to test the application SketchiXML extends a set of tools based on UsiXML, allowing to initiate the design process from the early design phase to the final concrete user interface for different platforms, with tools support for every stages

70Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Conclusion  Meet all the requirements elicited in the state of the art and particularly the Development of a scenario editor allowing to represent transition between screen.  During the sketching process, the possibility to instantly switch to a runnable version of the current UI would be useful. Indeed, all informal design tools providing code generation allow easy switching from design to run mode, while SketchiXML requires to invoke a third application.

71Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006 Thank you very much for your attention For more information and downloading, User Interface eXtensible Markup Language European network on Multimodal UIs SketchiXML home page