Nov Jason Hong and James Landay University of California Berkeley Group for User Interface Research

Nov Motivation Sketching is useful in many settings

Nov Creative / Design SILK Music Notepad Knight Elec. Cocktail Napkin Teddy PatchWork Survey of Informal Ink Apps Capture ZenPad NotePals Dynomite FiloChat Others Pegasus XLibris PerSketch Kramer's Patches Whiteboard Tivoli Flatland

Nov Example - Elec. Cocktail Napkin Gross and Do (UIST96)

Nov Example - Flatland Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)

Nov Common Features Pen input as ink or gesture Stroke is pen input from dragging Ink is a stroke that stays on-screen Gesture is a stroke that activates a command Recognize ink as objects Transformation / clean-up of ink Immediate / deferred processing of ink erase me

Nov Common Features Contain objects in addition to ink Selecting and moving objects Grouping of objects Layering of objects

Nov Related Work Commercial Software Support –Newton OS –PenPoint –Microsoft Windows for Pen Computing –Windows CE –Palm OS Problems –Form-based and handwriting interfaces –Not easily extensible for building informal ink apps

Nov Related Work Research Software Support –ArtKit (Henry et. al. 1990) –Garnet Pen Int.(Landay et. al. 1993) –Patches (Kramer, 1994) –OOPS (Mankoff et. al. 2000) Problems –Need more reusable libraries for handling and processing strokes –Need more extensibility so lots of kinds of informal ink apps can be built

Nov Problems Software infrastructure support for informal ink apps rudimentary Pen-based interaction techniques difficult to implement Similar to GUI development in 1980s –Lots of applications + interaction techniques –Need cohesive frameworks + toolkits

Nov Goals of SATIN Toolkit support for informal ink apps –Provide reusable mechanisms for handling + processing strokes –Separate mechanism from policy –Be extensible for new kinds of apps Reusable widgets for pens Distribute this toolkit for use

Nov Overview  Motivation  Survey of Informal Ink Applications  Two Applications Built with SATIN  SATIN Architecture Overview  Conclusions

Nov DENIM

Nov SketchySPICE

Nov SketchySPICE

Nov SketchySPICE

Nov Some Metrics SATINDENIMSketchy SPICE #source files #methods #classes #lines of code time1 person 18 months 3 people 3 months 1 person 3 days

Nov Overview  Motivation  Survey of Informal Ink Applications  Two Applications Built with SATIN  SATIN Architecture Overview  Conclusions

Nov Architecture Overview Java Core Classes Java Virtual Machine Java2D SATIN Swing Application SATIN relies on Java2D and Swing Applications use SATIN, can use Java2D and Swing

Nov Architecture Overview SATIN can be broken into twelve concepts Recognizers InterpretersScenegraph Stroke Libraries Widgets Views Rendering Transitions Clipboard Notification Command Events

Nov Recognizers Problem - many recognition techniques Recognizers let new recognition technologies be plugged into SATIN Given a stroke, return n-best list No actions taken SATIN comes with Rubine's Recognizer (implemented by Chris Long) Recognizer Cut Copy Undo

Nov Interpreters Problem – many ways of handling strokes Interpreters let new ways of handling strokes be plugged into SATIN e.g. straighten a line e.g. issue a command Ink Interpreter Gesture Interpreter erase me

Nov Interpreters and Recognizers Decoupled recognition (Recognizers) from actions (Interpreters) Interpreters can use recognizers Both are modules that can be plugged into apps built with SATIN InterpreterRecognizer Cut Cut Copy Undo

Nov Multi-Interpreters Problem – Hard to combine and extend existing interpreters Multi-Interpreters are a collection of interpreters + dispatch policy –e.g. dispatch to chain of interpreters –e.g. disable some interpreters on context Default Multi Interpreter Intrp AIntrp Z … Semantic Zoom Multi Interpreter Intrp B Intrp AIntrp Z … Intrp B

Nov Phrase (Patch) Scenegraph Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke … Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

Nov Stroke Dispatching Needed an approach that enabled: –Individual objects handle strokes on own –Priority to gestures over ink Default policy is top-down dispatch 1.Process stroke with its gesture interpreter 2.Re-dispatch stroke to one of its children 3.Process stroke with its ink interpreter 4.Handle stroke in object itself

Nov Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke

Nov Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke 1. Process stroke with gesture interpreter Sheet Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp …

Nov Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke

Nov Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Sheet Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp … 1. Process stroke with gesture interpreter

Nov Stroke Dispatching 2. Re-dispatch stroke to children Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)

Nov Stroke Dispatching 1. Process stroke with gesture interpreter Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)

Nov Stroke Dispatching 2. Re-dispatch stroke to children Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)

Nov Stroke Dispatching 3. Process stroke with ink interpreter Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch) Phrase Interpreter

Nov Stroke Dispatching 4. Handle stroke in object Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)

Nov Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke

Nov Example - Cut Shallow

Nov Example - Cut Deep

Nov Reusable Stroke Libraries Straighten Merge

Nov Reusable Stroke Libraries Split Simplify

Nov Widgets Pie Menus Pen PLAF Swing "Pluggable Look And Feel" Larger and clickable sliders Single-click file opener

Nov Conclusions SATIN - Java toolkit for informal ink apps –Extensible thru Recognizers, Interpreters, and Multi-Interpreters –Separates mechanism from policy for strokes –Offers reusable libraries for manipulating ink Reusable widgets for pens

Nov Download SATIN Download SATIN at

Nov Extra Slides

Nov DENIM

Nov DENIM

Nov Survey of Informal Ink Apps Elec. Cocktail Napkin Flatland Pegasus FiloChat Zenpad PerSketch Teddy PatchWork XLibris NotePals Dynomite SILK Tivoli Kramer's Patches Music Notepad Knight

Nov Architecture Overview Recognizers Interpreters Scenegraph Stroke Libraries Widgets Views Rendering Transitions Clipboard Notifications Commands Events SATIN can be broken into twelve concepts