1. SKETCH-BASED USER INTERFACE STUDY Presented By Jin Xiangyu Department of Computer Science and Technology Nanjing University June 2002

2. PART I: INTRODUCTION

3. The rise of the research issue of Human-Computer Interaction (HCI) The idea of this revolution is to bend computers to people’s way of interacting, not the other way around (Landay 2001) Human and computer, which one should be the center of computer assisted tasks? Computer-Oriented Human-Oriented 1.1. A Revolution is Undertaking Computers are becoming more and more powerful and easily available today

4. 1.2. Why Sketch-based User Interface? To write down designer’s improvisatory ideas by diagrams is very important for creative tasks. (3) Unsuitable for handled devices No area to accommodate so many stencils and buttons. Traditional menu/toolbar button-based user interface DemoDemo (1) Inefficient A three-step process. (2) Unnatural Leaving a sketch uninterrupted, or at least in its rough state, is key to preserving this fluidity (Hearst 1998). The Solution is “By Sketch”. Sketching with a pen is a mode of informal, perceptual interaction that has been shown to be especially valuable for creative design tasks (Gross 1996).

5. 1.3. Research focus: What Kind of Sketch- based User Interface We are Interested in? On-line VS Off-line ModeInputProcessSegmentationVectorizationStroke On-line f(t): R +  R×R InteractiveEasyNot requiredCursive Off-line2D-Boolean MatrixOne-stepDifficultRequiredFormal Three application level Sketchy symbol recognitionStroke-number and stroke order free Not very strictly defined 2 Sketch-based image retrieval Handwriting recognition Example Totally free Generally agreed among users Drawing ApproachSymbol SetLevel Undefined Strictly defined 3 1 Our research focus

6. 1.4. Three Designing Principles: Humanistic, Intelligent, Individualized How to make the UI to be humanistic? An graphics inputting user scenario is proposed, which employs an interactive sketching-recognition-rectification process in one-fluent-step. How to make the UI to be intelligent? On-line graphics recognition is employed to predict user’s original intention. How to make the UI to be individualized? SVM-based incremental learning is employed to adapt different user in shape classification. These three characteristics are harmoniously combined in our prototype system —Smart Sketchpad.

7. PART II: HIGHLIGHTS

8. 2.1. Critical Technique 1: User Scenario for Graphics Inputting Employing one interactive, fluent sketching-recognition-rectification process instead of three split ones. Step 1: Sketching Recognize and Regularize The suggested candidate objectsStep 3: Clicking on the intended object and replace the strokes with the very object with proper parameters Step 2: Sketching Recognize and Regularize

9. 2.2. Critical Technique 2: On-line Graphics Recognition Strokes Primitive ShapeComposite Graphic Object Primitive Shape Classification and Regularization Composite Shape Recognition Sketching (user) : Decompose Composite Graphic Object Recognition (computer) : Assemble Primitive ShapeStrokes

10. 2.2.1. Primitive Shape Classification and Regularization Strokes Primitive Shape Classification & Regularization Primitive Shapes The input stroke PreprocessingShape Classification Quadrangle Shape Fitting The fitted shape Shape Regularization The regularized shape The User Intended Shape The User Sketchy Shape By Vertex Combination

11. Experimental Results Shape Classification Precision for 1367 samples User1 0.75 User2 0.35 The optimal thresholds are different

12. Experimental Results Shape Regularization Results Inner-Shape Regularization The results of primitive shape classification and regularization

13. 2.2.2. Composite Graphic Object Recognition A “ Partial ” “ Structural ” Similarity Assessment Strategy is Proposed In order to suggest the user in an early stage, the system should recognize graphic object in an incomplete form.

14. 2.2.2. Composite Graphic Object Recognition A “ Partial ” “ Structural ” Similarity Assessment Strategy is Proposed The similarity assessment strategy should not only invariant to shifting, rotation, mirroring, but also should invariant to inner distortion.

15. The Proposed Approach The Source ObjectThe Candidate Object Spatial Relation Graph (SRGs) The computational complicity is P n m. Conditioned Partial Permutation Algorithm L1L1 L2L2 L3L3 L4L4 P1P1 P2P2 P3P3 P4P4 P5P5 Graphic Primitive Extraction Line-segments, arc-segments, and ellipses/circles

16. Performance Evaluation Query Generating by adding noises and eliminating some parts. Experimental results show that our approach can achieve good performance with noises for incomplete objects, and our approach is also invariant to shifting, rotation, mirroring, and inner distortions. When the user draws 80% of his/her intended object (for users may miss some parts of the object inadvertently) with 10% distortion (this is similar to noises in real user drawing situation), R6 is nearly 90% (averagely of 304 graphic objects). The Original ObjectAdding NoisesEliminating some parts The Generated Query

17. 2.3. Critical Technique 3: User Adaptation User adaptation is a classical problem in user interface study. Many pattern recognition problems are user specific, for users’ handwritings, drawing styles, and accents are different. Rule-based feedback may yield “conflict” results due to its intrinsic deficiency, which may lose its general performance when it adapts to a specific user further. Question: A triangle or a quadrangle? An ambiguous case SVM-incremental learning are introduced into the user adaptation problem of shape classification.

18. 2.3.1. Questions Four questions need to be solved: (1)Whether SVM-based Incremental learning can overcome “conflicts”? (2)What is the advantage of Incremental leaning compared with repetitive learning? (3)Which one is better, Syed’s or Xiao’s? (4)Which structure is better, one-against-one and one-against-all? 2.3.2. Experiments Experimental Environments: Feature Extraction (20-dimensional vector) by turning function Virtual Sample Generation (with 40 samples each) 40 incremental training sets and two test sets are created Training time, open-test precision, closed-test precision are tested for different algorithms and structures.

19. 2.3.3. Answers (1)SVM-based incremental leaning can overcome “conflict” (2)Incremental learning is much faster than repetitive learning without loss of precision (3)Syed’s algorithm is better than Xiao’s (4)One-against-one structure is much faster than one-against-all in our environments Theoretical analysis and experimental results both show

20. PART III: THE SMART SKETCHPAD SYSTEM

21. 3.1. System architecture of Smart Sketchpad

22. 3.2. The Sketch-based User Interface of Smart Sketchpad The Sketching Area The just inputted shape are recognized and regularized The candidate shape are shown Part of the intended object are sketched Candidate object name and their similarities are shown Candidate object list Inputting two graphic objects and then delete them Demo

23. 3.3. UI Evaluation 10 different subjects are required to draw the following two diagrams with traditional UI and the sketch-based UI. There are 304 objects listed in 26 stencils (with 12 each) for traditional UI. There are 6 objects can be shown in the Smart Toolbox for sketch-based UI. Diagram 1 Diagram 2 A demo of inputting Diagram 2 by sketch

24. Drawing Time for Different Sketches Under Different UIs Averagely, the sketch-based UI is 22.4% and 42.9% more efficient than the traditional toolbar button-based UI for sketch1 and sketch2, respectively. The ultimate comments of all users unanimous agree that they’d like to choose the sketch-based UI instead of the traditional one.

25. PART IV: SUMMAY

26. Algorithm Level Solution Level System Level (1) Agglomerate Point Filtering, Vertex Combination, Shape Fitting, Shape Regularization (2) Conditioned Partial Permutation (3) Comparison of SVM-based Incremental Learning Algorithms and Structures (1) Interactive graphics inputting user scenario (2) On-line graphics recognition: primitive shape classification and regularization; composite shape recognition. (3) SVM-incremental learning for user adaptation in shape classification problem. (1) A prototype system for conceptual/schematic designing tasks is implemented. (2) User evaluation between the traditional and sketch-based UIs is performed.

27. Future Works (1) How to perform stroke segmentation? (2) How to cut down the computational cost and improve the recognition precision? (3) How to make the system learn aggressively?

28. THANKS