1. 1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Trainable Sketch Recognizer for Graphical User Interface Design Adrien Coyette 1, Sascha Schimke 2, Jean Vanderdonckt 1 & Claus Vielhauer 2 1 Université catholique de Louvain (UCL) School of Management (IAG) Information Systems Unit (ISYS) Belgian Lab. of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi 2 Otto-von-Guericke Universität Magdeburg Fakultät für Informatik Institut für Technische und Betriebliche Informationssysteme (ITI) http://wwwiti.cs.uni-magdeburg.de/

2. 2 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Considerable importance of user interface in application development (50% to 70 % of the application code [Myers and Al, 2000] ) Apparition of many high fidelity graphical user interfaces editors on the market Motivations

3. 3 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007  But designing the right User Interface (UI) the first time is very unlikely to occur iterativeincomplete  instead, UI design is eminently open iterative and incomplete Motivations

4. 4 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Paper ( or whiteboard) used most of the time:  familiar and unconstrained approach  fast to learn and quick to produce  focus on basic structural issues instead of unimportant details  it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,  can be performed collaboratively between designers and end-users Motivations

5. 5 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Motivations: empirical findings  UI Drawing or sketching increases participatory design and subjective satisfaction with respect to UI editors [Landay & Myers, 1995, Plimmer & Apperley, 2004, Buxton, 2005]  Equal usability problems found for a sketched UI than with an edited one [Virzi et al., 1996]  Explanatory power of a sketched UI does not decrease with respect to edited one; tool support is preferred over paper prototyping [Walker et al., 2002 ]  A sketched UI delivers same quantity and quality of outputs than an edited one [Sefelin et al., 2006]  UI sketching fosters alternative designs and communication [Tohidi et al., 2006]  Supports throw-away prototyping due to low cost [Schneider & Petrie, 2006, Bailey & Konstan, 2003]

6. 6 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007  The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations  Such tools would extend the advantages provided by sketching techniques (easily creating, deleting, updating or moving UI elements). hybridapproaches  Apparition of hybrid approaches, combining the best of the hand-sketching and computer assisted user interface design Related Works

7. 7 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 State of the Art: Eliciting requirements  17 different software and approaches surveyed  Sources: scientific literature and commercial software  Comparative analysis relying on consistent analysis grid  Tool identification (7)  Install and first use (4)  Tool functions (8)  Shape recognition (4)  Shape interpretation (7)  UI editors (9)  Applied to the same case study (eCommerce app.)

8. 8 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 DENIM (James A. Landay, James Lin, Mark W. Newman, Jason I. Hong) Related works

9. 9 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Advantages: Support for scenario-based design Several levels of granularity Good documentation Toolbox of generic widgets Mature product, based on experimental testing Zooming facility from local design (e.g. a web page) to a global design (e.g., a portion of a web site or n entire web site) Storyboarding facilities based on patternsShortcomings: No shape recognition and interpretation, thus loosing the effort No code generation No preview mode Only dedicated to web sites Related works

10. 10 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 JavaSketchIt (Joaquim A. Jorge, Manuel João Fonseca, Anabela Caetano, Néri Goulart ) Motivations / Related works

11. 11 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Advantages: Performance (speed and accuracy) Multi-stroke gestures Recognizes rotated shapes Computationally light Open source Requires standard and freely available libraries Shortcomings: Mono-window No scenario editor Only generates java Limited widget set Shape interpretation can only take as input a construct made of maximum two vectorial shapes Motivations / Related works

12. 12 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007  Avoidance of Effort loss  Language neutrality  Ease of use (naturalness)  Flexible processing  Robust scenario editor  Large conceptual coverage  … Motivations / Related works

13. 13 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML

14. 14 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML

15. 15 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Shape Recognition Recognition engine: Cali library  A fast, simple and compact  A fast, simple and compact approach to identify Scribbles (multi- stroke geometric shapes) temporal adjacency, Fuzzy Logic and geometric features  Combines temporal adjacency, Fuzzy Logic and geometric features to classify scribbles.  Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap. recognized independently of changes in rotation, size or number of individual strokes  Shapes are recognized independently of changes in rotation, size or number of individual strokes 94%  The recognizer has a recognition rate of 94% and a fast response

16. 16 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Shape Recognition All widgets were based on a set of shape primitives Not extendable Require a delay between each strokes Limits the number of representation

17. 17 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Shape Recognition Trainable Gesture Recognizer  Handdrawn input (sketch) is superimposed with a grid  Each sampling point is mapped to its closest grid node  For adjacent nodes, a stroke direction (out of 8 different directions) is taken  A string is generated from direction coding of input  Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances

18. 18 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Shape Recognition Distance is shortest sequence of edit commands that transform s to t Simplest set of operations: Copy character from s over to t Delete a character in s (cost 1) Insert a character in t (cost 1) Substitute one character for another (cost 1) Levenstein distance

19. 19 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Shape Recognition

20. 20 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 SketchiXML: Grammar

21. 21 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Conclusion / Future Works Conclusion  an innovative contribution to the domain of sketch based design tools: a trainable multi-strokes gesture recognizer  Improved the performance of SketchiXML Future works  evaluating other feature types to describe the gesture shape, e.g. triangular grids  Improving the performances of the algorithm, when running with a large set of representations  Conduct a systematic comparative test with other shape recognition algorithms

22. 22 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs http://www.usixml.org/index.php?view=page&idpage=29 SketchiXML home page