1. SATIN: S ketch And T ransformational IN frastructure Jason Hong UC Berkeley Group for User Interface Research Project Status Report - 07/22/1999

2. Motivation More and more apps are using pens and some form of sketching These apps are usually –built from scratch –not extensible or reusable Pilot Elec. Cocktail Napkin Dynomite Pegasus FiloChat Zenpad XLibris NotePals Flatland SILK Tivoli Kramer's Patches

3. Motivation Architecture of windowing systems optimized for keyboard and mouse –Need to explore principled architectures for alternative input –Current state is analogous to windowing systems in the early 1980s

4. High-Level Research Goals Create extensible toolkit supporting creation of pen and sketch-based apps Explore architectures for pen-based applications Explore new interaction techniques for using pens and for sketching Explore new event models for other input devices

5. Overview Motivation Envisioned Applications Features and Current Status High-Level Architecture Future Development Feedback

6. Envisioned Applications SILK DENIM

7. Envisioned Applications PosterBoard –Create posters using DigitalDesk

8. Envisioned Applications PatchBoard –Multi-device whiteboard

9. How SATIN Fits Into the Big Picture SATIN Swing Core Classes Java Virtual Machine Application All versions of Java have these parts Users may already have these parts or may have to download dynamically The Required Berkeley CS Diagram

10. What SATIN Will Provide Scenegraph and rendering support –GraphicalObjects  Objects with state, behavior, and view  State and behavior defined thru interpreters  Multiple views –Transformations  Per GraphicalObject, Rotate / Scale / Translate –Styles  Translucency, fill colors, line width, etc  Just set it - automatically used!  Can define styles in text files  Cascading Styles e.g. selected objects or greyed-out objects Find

11. What SATIN Will Provide Scenegraph and rendering support  Add, remove, group, move, etc  Pick GraphicalObject at location  Layering of objects  Automatic animation for smooth transitions  Semantic zoom

12. What SATIN Will Provide Patches  Interpreted regions of space to help organize, process, and manipulate sketches Step A Step B Step C

13. What SATIN Will Provide Built-in support for Undo / Redo / Macros  Command pattern scheme  Automatic time-indexing of operations  Multiple streams and histories of events Integration with Java Swing  Cut / copy / paste  Standardized Drag and Drop  Export as GIF / XML  Allow use of Java widgets  Be used as a Java widget

14. Current Status Recognizers  Rubine's Recognizer Interpreter  Graffiti Interpreter Pie Menus Performance  Minimal damage-redraw Needs some more optimization though

15. Current Status Basic Debugging Support  Debugging Window  Graphical Bounds Properties file  System initializations Sheet colors, location of data files, etc Documentation  What you are reading  LOTS of Javadoc

16. Overview Motivation Envisioned Applications Features and Current Status Future Development Feedback High-Level Architecture

17. Short-term Future Plans More Patch Operations –pull up, push down, dissolve, auto-arrange, Multiple Views Advanced Dispatching –"tolerant" dispatching –top-to-bottom dispatching Streams and histories –Object histories or system histories? Activators –If "is matrix" then load matrix interpreter

18. Feedback? What do you need to make this work for you? What do you like? What do you not like?

19. Overview Motivation Envisioned Applications Features and Current Status Future Development Feedback High-Level Architecture

20. GUIR Object Notation Car Vehicle JComponent Pinto Wheel 14 Gadget 1n This color means classes already provided by Java The up arrow points to the parent class Each instance of Car has four instances of Wheel Each instance of Pinto has n instances of Gadget. Each Gadget has a reference back to the Pinto it belongs to. Semi

21. GraphicalObject (GOb) GraphicalObject Collection GraphicalObject Group PatchSheet JComponent Watchable Watcher Simplified Overview of Class Architecture Interpreter Interpreters Timed Stroke

22. Runtime Architecture Java2D Drawing Stroke Assembler Java Event Subsystem Rendering Subsystem Dispatching SheetGraphicalObject Group GraphicalObject (GOb) Interpreter Patch Commands Command Queue (Undo, Redo, Macros) ClipboardSelected GObs Command Objects

23. Watchable GraphicalObjects in Depth Transform Style GraphicalObject Group Interpreter Mediator Serializable Cloneable Interpreter Interpreters GraphicalObject (GOb) Shape SingleStroke Listener Progressive Stroke Listener 1n Bounding Points (Polygon)

24. The Lifecycle of a Stroke (Current) GraphicalObject Group GraphicalObject (GOb) Sheet Interpreter Mediator Interpreter Patch Mouse Events get forwarded here until the full stroke is completed. The Stroke Assembler then dispatches to the Sheet. Sheet determines if the stroke should be redispatched to another GraphicalObject onSingleStroke() 1 Stroke Assembler 2 onSingleStroke() 3 handleSingleStroke() 4 The deepest object tries to handle the stroke first.

25. The Lifecycle of a Stroke (Current) Sheet Stroke Assembler onSingleStroke() mouseDragged() Graphical Object GraphicalObjectGroup / Patch handleSingleStroke() GraphicalObjectGroup / Patch onSingleStroke() isHandled? if (!isHandled) { isHandled = handleSingleStroke(); } return (isHandled) dispatchee = getDispatchee() if (dispatchee != null) { isHandled = dispatchee.handleSingleStroke(); } if (!isHandled) { isHandled = handleSingleStroke(); } return (isHandled) A B AB B A A

26. The Lifecycle of a Stroke (Future) Sheet Stroke Assembler dispatch() mouseDragged() Graphical Object GraphicalObjectGroup / Patch handle() GraphicalObjectGroup / Patch dispatch () isHandled? Every GraphicalObject has nine methods: dispatchNewStroke() processNewStroke() handleNewStroke() dispatchUpdateStroke() processUpdateStroke() handleUpdateStroke() dispatchSingleStroke() processSingleStroke() handleSingleStroke() process () dispatch() handle() isHandled?handle()

27. GraphicalObject Coordinate Spaces Every GraphicalObject has three coordinate spaces The top-left corner of this stroke is at (246,335) in absolute or screen coordinates. The top-left corner of this stroke is at (91,59) in relative or parent's coordinates The top-left corner of this stroke is at (0,0) in local or self coordinates In local coordinates, every GraphicalObject will have it's top-left corner at (0,0)

28. GraphicalObject Coordinate Spaces In absolute coordinates, the top-left corner of the Sheet is always (0,0). If you turn Graphical Debugging on, the coordinates displayed are absolute This is NOT (0,0) for the Patch. The transform was applied to the Sheet, changing absolute coordinates, but not anyone's local coordinates. This is (0,0) for the Patch Here, we have two screen shots. The one on the left has no transform. The one on the right has a rotating transform on the Sheet.

29. GraphicalObject Coordinate Spaces How it works internally… SheetTransform (txA) PatchTransform (txB) GraphicalObjectTransform (txC) Absolute Transform = [txA][txB][txC] Relative Transform = [txC] Local Transform = [1] Absolute coordinate methods: getAbsoluteTransform() getAbsolutePosition() getAbsoluteBounds() getAbsoluteBoundingPoints() getAbsoluteWidth() getAbsoluteHeight() setAbsoluteBoundingPoints() Relative coordinate methods: getTransform() getPosition() getBounds() getBoundingPoints() getWidth() getHeight() setBoundingPoints() Local coordinate methods: getLocalTransform() getLocalPosition() getLocalBounds() getLocalBoundingPoints() getLocalWidth() getLocalHeight() setLocalBoundingPoints()

30. Recognizers and Interpreters in Depth Internal Data Structure Helpers Recognition Data Struct(s) Ink Clustering Algorithms (Spatial, Chronological) Recognizer StrokeClassification Interpreter Recognizer Stroke Classification GraphicalObject (GOb) Handled? App-Specific Code Internal Data Structures Interactor Tree Probabilistic Tree Recognizers Rubine Recognizer Neural Net Recognizer Progressive Recognizer Single-Stroke Recognizer Interpreters Progressive Interpreter Single-Stroke Interpreter Graffiti Interpreter Rectilinear Interpreter