1 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 A Methodological Framework for Multi-Fidelity Sketching of User Interfaces Adrien Coyette Université catholique de Louvain (UCL) Louvain School of Management (IAG-LSM) Information Systems Unit (ISYS) Belgian Lab. of Computer-Human Interaction (BCHI)

2 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

3 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Motivations: UI importance User interface determines how easily a user may control underlying functions of a computer program A program equipped with powerful functionalities and low quality user interfaces may be under- exploited or misused In an interactive application, the UI is probably the portion which affects the most the general acceptability of the system by end users  On the average, 48% of the code written for the 69 applications studied was devoted to the user interface portion [Nielsen 93; Myers & Rosson 92]

4 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Motivations: UI development life cycle Software plans and requirements Validation Product design Verification Detailed design Verification Code Unit test Integration Product verification Operations and maintenance Revalidation Implementation System test System feasibility [Boehm88] UI Design occupied 45% of the total amount of time devoted to each activity 50% 37% Mean = 44%

5 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Motivations: methods  As in software engineering, HCI community wants to develop a well-structured and step-wise method for developing user interfaces.  The systematization and the reproducibility found in software engineering methods however cannot be transferred straightforwardly in HCI  The development life cycle remains inherently:  Open, Ill-defined, Highly iterative  Interaction is of higher complexity than algorithms Sumner et al. [Sumner et al., 1997; Wegner, 1997]

6 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Motivations: iterative design  Designers often consider that informal techniques based on mock up is the best alternative for the early design phase.  It allows to cope with the fact the such process is highly iterative in 3 stages Norman, 1996 [Norman, 1996]

7 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Motivations: empirical findings  UI Drawing or sketching increases participatory design and subjective satisfaction with respect to UI editors [Landay & Myers, 1995, Plimmer & Apperley, 2004, Buxton, 2005]  Equal usability problems found for a sketched UI than with an edited one [Virzi et al., 1996]  Explanatory power of a sketched UI does not decrease with respect to edited one; tool support is preferred over paper prototyping [Walker et al., 2002 ]  A sketched UI delivers same quantity and quality of outputs than an edited one [Sefelin et al., 2006]  UI sketching fosters alternative designs and communication [Tohidi et al., 2006]  Supports throw-away prototyping due to low cost [Schneider & Petrie, 2006, Bailey & Konstan, 2003]

8 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

9 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Prototyping: when? Early design Late design Low fidelity paper prototypes Content: Mainly Presentation Use: Exploration, Communication Medium fidelity prototypes Content: Presentation, content, basic navigation Use: Simulation, refinement, user testing… High fidelity prototypes Content: Presentation, navigation, functionalities, content Use: Final specifications, marketing, documentation A paper based prototype A PowerPoint prototype The final user interface with few functionalities

10 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Prototyping: sketches Paper prototypes –Familiar and unconstrained approach –Fast to learn and quick to produce –Focus on basic structural issues instead of unimportant details –Support collaboration between designers and end-users –… Purpose –Early design phase –Get a sense of the user’s needs and goals –Supports brainstorm competing representations –…

11 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Prototyping: Limiting prototype functionality Vertical prototypes –Includes in-depth functionality for only a few selected features –Common design ideas can be tested in depth Horizontal prototypes –The entire surface interface with no underlying functionality –A simulation; no real work can be performed Qcenario –Scripts of particular fixed uses of the system; no deviation allowed Vertical prototype Scenario Horizontal prototype Full interface [Nielsen 1993]

12 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Prototyping: Integrating prototypes and products Throw-away –Prototype only serves to elicit user reaction –Creating prototype must be rapid, otherwise too expensive Incremental –Product built as separate components (modules) –Each component prototyped & tested, then added to the final system Evolutionary –Prototype altered to incorporate design changes –Sometimes becomes the final product

13 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

14 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 State of the Art: Computer Assisted Prototyping Several levels of granularity Support for scenario- based design Good documentation No shape recognition and interpretation No code generation  No preview mode DENIM [Lin,2000]

15 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 State of the Art: Computer Assisted Prototyping Performance (speed and accuracy) Multi-stroke gestures Recognizes rotated shapes Interpretation in Java Only generates Java No scenario editor Limited widget set Mono-window JavaSketchIt [Caetano,2002]

16 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 State of the Art: Eliciting requirements  17 different software and approaches surveyed  Sources: scientific literature and commercial software  Comparative analysis relying on consistent analysis grid  Tool identification (7)  Install and first use (4)  Tool functions (8)  Shape recognition (4)  Shape interpretation (7)  UI editors (9)  Applied to the same case study (eCommerce app.)

17 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 State of the Art: List of final requirements  Avoidance of Effort loss  Well defined editing functionalities  Language neutrality  Robust recognition  Large conceptual coverage  Recognition and process flexibility  Design history  Expressive scenario editor  Ease of use (naturalness)  Preview (Run-mode)

18 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

19 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Demonstation

20 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Shape Recognition

21 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Shape Recognition Recognition engine: Cali library  A fast, simple and compact  A fast, simple and compact approach to identify Scribbles (multi- stroke geometric shapes) temporal adjacency, Fuzzy Logic and geometric features  Combines temporal adjacency, Fuzzy Logic and geometric features to classify scribbles.  Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap. recognized independently of changes in rotation, size or number of individual strokes  Shapes are recognized independently of changes in rotation, size or number of individual strokes 94%  The recognizer has a recognition rate of 94% and a fast response

22 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Shape Recognition Trainable Gesture Recognizer  Handdrawn input (sketch) is superimposed with a grid  Each sampling point is mapped to its closest grid node  For adjacent nodes, a stroke direction (out of 8 different directions) is taken  A string is generated from direction coding of input  Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances

23 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Shape Recognition Distance is shortest sequence of edit commands that transform s to t Simplest set of operations: Copy character from s over to t Delete a character in s (cost 1) Insert a character in t (cost 1) Substitute one character for another (cost 1) Levenstein distance

24 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Shape Interpretation

25 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 SketchiXML: Grammar

26 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

27 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Methodologies: framework Prototyping types User interface Control layer Functional core Complete interactive application Horizontal prototypeVertical prototype User interface Control layer Functional core Complete interactive application Diagonal prototype [Nielsen93]

28 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Methodologies: framework Prototyping types User interface Control layer Functional core Complete interactive application Presentation Global navigation Local navigation

29 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Methodologies: framework Prototyping types Control layer Functional core Présentation Global navigation Local navigation Presentational prototype first Control layer Functional layer Presentation Navigation globale Local navigation Global navigation prototype first Control layer Functional core Présentation Navigation globale Navigation locale Local navigation prototype first

30 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Methodologies: Support by fidelity High fidelity prototype : GrafiXML Medium fidelity prototype : VisiXML Low fidelity prototype: SketchiXML

31 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Methodologies: Application to development methodologies Analyst claimed in the past that prototyping should be considered as an alternative to the systems development life cycle… More recent analysis advocate the use of prototyping as a part of the traditional systems development life cycle  Existing methodologies tend to integrate prototyping as part of the approach Based on the framework we are thus able to support effectively the current development methodologies. SketchiXML permit to involve end users without pre-requisite [Kendall 97]

32 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Best practices: Collect information Develop simple prototype Discuss with the end-users Test interaction between end user and prototype Adapt the fidelity level Avoid polished prototype Iterate Applied to the latest methodologies  Applied to the latest methodologies [Meyer 2005] Methodologies: Application to development methodologies

33 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

34 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: Widget catalogue Building the widget catalogue

35 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: widget catalogue

36 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: User testing of fidelity Method: 12 participants Wacom graphic tablet 12 widgets randomly selected 4 fidelity levels Repeated twice Factors observed: Time required Number of errors Number of delete operations

37 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: User testing of fidelity Results: Fidelity has no impact on the sketching of individual widget (time or error) User profile influence the widget sketching time (non UI designers performed even better !) Proved learning effect over time

38 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: Performances Widget Recognition rate Sketching Time Q1Q2Q3Q4totalQ1Q2Q3Q4total Button Checkbox Combox Label Listbox Picture Progress Radio Slider Textarea Textfield Toggle Total ,623071,193222,152637,613542,39

39 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Surveys: Representations

40 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Presentation plan  Motivations  Prototyping  State of the Art  SketchiXML  Methodologies  Surveys  Conclusion

41 Private Ph.D. defense – Louvain-la-Neuve, June 25th, ) Fundamental Notion of fidelity level (low, medium, high) in multi-fidelity prototyping Graphical definition of widgets in terms of representations Formal definition of representations in a grammar Gesture recognition algorithm based on Levenstein distance 2) Methodological Multiple prototype types (diagonal=vertical+horizontal) and paths Intertwining of presentational and navigational prototyping Integration within various development life cycles Transition between the fidelity levels (within SketchiXML and other tools based on UsiXML) at Concrete User Interface level Conclusion: contributions

42 Private Ph.D. defense – Louvain-la-Neuve, June 25th, ) Empirical User and designer acceptance of widget representations User testing of sketching tool Complexity of widget representations 4) Implementation Implementation of SketchiXML (50000 LOC) Multi-agent shape recognizer –8 basic shapes (i.e., triangle, rectangle, cross, line, wavy line, arrow, ellipse, and circle) –6 basic commands (i.e., undo, redo, copy, paste, cut, new window) –27 widgets Trainable gesture recognition engine Multi-fidelity support by smooth transition between (none, low, medium, high) Multi-agent interpreter All 10 requirements addressed (partially or completely) Conclusion: contributions

43 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Conclusion: future work Extending the coverage of sketching artifacts Improving the Text Divider Tuning the Recognition Engine more extensively Support for Multi-windowing Design Augmenting the Support for Design Memory Extending to other domains than Computer Science General Improvement of sketching facilities

44 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Conclusion: example of extension Sketching Physical User Interfaces with transition between digital and physical worlds

45 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 For more information and downloading, User Interface eXtensible Markup Language European network on Multimodal UIs Special thanks to all members of the team! This work has been conducted in the context of the following projects Projet WIST 1 ReQuest Rapid prototyping of e-commerce applications

46 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007 Thank you very much for your attention! Questions?