Sketching Interfaces for Specification

Readings James A. Landay, Brad A. Myers, “Sketching Interfaces: Toward More Human Interface Design”, IEEE Computer, v34, n3, pp , Bailey, B.P., J.A. Konstan, and J.V. Carlis, "DEMAIS: designing multimedia applications with interactive storyboards", Proceedings of the International Conference on Multimedia, 2001, Ottawa, pp

Supporting Early Design Key tasks being supported –Design space exploration Medium for expression and “visual thinking” –Increase concreteness –Providing inspiration and points of departure Understanding the implications of design choices –Communications Vehicle for talking about design options with others

Sketching is Good Almost everyone starts in early stage with very rough pencil and paper sketches –Rough ideas quickly –Uncertainty Uncertainty and ambiguity is important to have at this stage (important to creativity) Can’t have the details before you have done the design –Fluidity is critical Investment must be very low Changes must be very easy Must stay “light on your feet” and not get solidified into small range of alternatives too early

Need ability to test / operate as early as possible But also need to be able to evaluate the real meaning / consequence of things  Must make them “operable” or executable at some level to get a real understanding –Big win Herein lies the hard problem: Ambiguity and uncertainty are very antithetical to being able to “execute” something

Getting past the divide Being able to “execute” something which is ambiguous and only partially defined –Need to infer intent –Need sensible interpretation of all partial specifications –Hard: probably need to limit domain Also need graceful path from early/ambiguous to later/precise (“real”) –Would rather not be forced to “throw it away” after a certain point

SILK Sketch –On digitizing tablet rather than paper Interactive rather than static setting Turn sketches into something that has enough structure and meaning that rich computational things can be done with it Try to get fluidity benefits of pen but add a lot more Storyboard –Used as central mechanism for specifying behavior –Limited, but well targeted (and already familiar to many) Transition to real widgets –Gives (some) “upward path” for results

SILK modes Sketch Annotate Decorate Run Also use of pen mode –Special button on pen for command gestures Modes help a lot with recognition

SILK interfaces Sketches –Window for sketching (specifying and placing) interactive components (widgets) –Screen at a time or dialog box at a time Sort of assuming static layout and somewhat moded style for result Storyboard –Window which contains collected copies of sketched screens –Place to specify behavior

Sketching widgets Attempt to recognize meaningful objects –Key to executability Make recognized sketches into operating widgets –Still appear as sketches Later transform into “real” widgets

Widget set Seven widgets in fixed set –Menu bar –Scrolling window –Palette –Button –Radio button –Check box –Text Field Plus rows and columns (“panels”) Fixed set shows up as a significant limitation in user tests But supporting an arbitrary set introduces big programming issues

Widgets recognized from primitive components Four primitive gestures recognized –Rectangle –Squiggly Line –Straight Line –Ellipse All single stroke –May require slight adjustment by user –Makes life a lot easier for recognizer No segmentation Rubine recognizer

Recognition Rubine recognizer –Single stroke –Feature based E.g., number of inflection points, angles, pt-pt distances, etc… –Not person independent Trained w/ examples Also feedback during use (via corrections made)

Recognition Steps –Individual component recognition –Spatial relationship analysis –Widget inference –Look for higher level groupings –Feedback –Possible user correction

Recognition steps Spatial relationship analysis –Relationships recognized Contains / is contained by Near –Left, right, above, below Vertical or horizontal sequence –(of the same type)

Recognition Steps Rule based widget inference  –Pattern Look for certain configurations of components –Rectangle contains rectangle and outer has very vertical aspect ratio  scroll bar –Body Compute confidence from details of matched parts Provide code for operating as a widget

Recognition Steps Look for “panels” –Horizontal or vertical groupings of widgets Proximity Alignment –Must all be the same type –Looks to create new panels or add to existing after each widget recognition

Recognition Steps Feedback –Recognized widgets turn purple (Not clear what feedback for groupings was) Usability problem uncovered in evaluation: Can’t tell which widget was recognized (without looking at control panel), but if you don’t get it right, correct panels don’t work

Recognition Steps  Biggest usability problem will turn out to be recognition errors User correction –Command gesture for “next best guess” Users can also give “hints” via selections –Explicit change in control panel –If no widget found control panel has “New guess” button to tell system to “look harder”

Also have recognition of command gestures Recall: special “command” button on pen –Very common approach All structured as a single stroke DeleteGroupUngroupNext Guess Insert Text

Specifying behavior via storyboards Can place individual screens (collections of ink, widgets, etc.) into storyboard view –Copy/paste then edit All behavior specified as navigation in story board –Specified by drawing a line (in storyboard editor) from either widget or background to a new scene. –In run-mode click on that object means “goto …”

Storyboards

Annotation and decoration Decoration mode –Basically turns off the recognizer –All strokes just in as unrecognized ink Annotation mode –Attaches ink and text to whole screen or individual widgets –Annotations can be turned on and off

Run mode Interface (with same sketched appearance) runs –Scroll bars slide, etc. –Buttons, etc. cause movements in storyboard May not look that way to the user (e.g., single change between boards) Also supplies some debug help –Storyboard highlights current and object which caused last last transition

Transition to “real” interfaces Can ask for sketched widgets to be turned into “real” widgets –VB or Garnet Still need to add: –Precise alignments –Color –Text labels –Actual icons images, etc –Real action code!

Experience & usability Overall seems very positive Biggest issue: recognition errors Recognition rates: Edit gestures:89% Primitives:93% Widgets:69%  not going to work End analysis:  Recognition probably worse than a more explicit strategy

Experience & usability Second big issue: fixed widget set –Can do some things, but support falls way off for “non-standard” interactions –A lot of effort to work around with inadequate tools  Hard problem because taking on large part of “the programming problem” may hurt simpler and more common aspect  Interesting research issues here

Demais System New (but related) domain: Multimedia presentations –Time based media (video and audio) –Timing and synchronization behavior now the tricky part Understanding design choices requires understanding how the detailed timing and sync will play out Contribution is in specification of behavior  A more complex programming task (but still limited)

Demais interface components Again, storyboard based –Single screen (layout) editor –Narration editor Deals with audio narration –Storyboard editor –Also “multi-view” editor Collections of screens, storyboards, etc. that are useful to pull up and manipulate together Sort of a “scrapbook” manager

Entities system deals with in a layout screen Plain text objects –Interpreted and uninterpreted –(Can do synch spec in scripting language) Ink strokes –uninterpreted Recognized objects Behavioral ink strokes Visual language icons Annotations

Recognition Very simple for sketched object recognition –Only recognizes two things (Rubine algorithm again) Rectangle –Indicates “content item” (still or video) –Tap to get file browser to load actual or simulated content Behavior stroke –Single line from something to something –Otherwise treat as uninterpreted ink Text recognition –Tries to parse as script text –If fails, uninterpreted

Narration editor Can insert “synchronization markers” in text –Points to synchronize other stuff against System does text-to-speech Also supports recorded speech but not clear how sync points are specified –Probably like video (position of slider when behavior attached), but not explicitly stated

Narration editor To synchronize, –e.g., image appearance with a point in the narration sketch a (behavior) stroke from the sync point to another storyboard element

Visual language for behaviors Behavior strokes are between objects –Creates implicit parameter and subject from source and destination of the stroke Strokes annotated to specify their actual behavior –Triggering event (from source) –Action to perform (at destination)

Behavior icons Events “T”s replaced by numbers –For sync: use current time of object –For elapsed: user prompted

Action icons

System picks default event & actions based on context

Can replace defaults Edit by tap-and-drop from palette –Better than drag-and-drop with pen Can also use text annotations in scripting language for specifying behaviors –English like (details not discussed here) –Infers referents (“this”) –Doesn’t seem to fit well