1. stanford hci groupoct 21, 2008 Björn Hartmann Loren Yu, Abel Allison, Yeonsoo Yang Scott R. Klemmer Design as Exploration Creating Interface Alternatives through Parallel Authoring and Runtime Tuning

2. 2 Exploration of alternatives is central to design.

3. 3 Prototypes for the Microsoft mouse From Moggridge, Designing Interactions, Ch2 MAP THE DESIGN SPACE

4. 4 From Design Secrets: Products 2 CommunicatE

5. Tohidi et al, CHI 2006 TEST 5

6. How can tools support the creation of user interface alternatives? 6 Color picker #1 Color picker #2 Color picker #3

7. “[D]esigners frequently wanted to have multiple designs side-by-side [...] However […] there is no built-in way in today’s implementation tools to have two versions of a behavior operating side-by- side.” Myers et al., VL/HCC 2008 7

8. Adobe Flash Arduino / Processing Adobe Dreamweaver Xcode (iPhone) Interaction Designers Write Code 8

9. Programming requires working with two representations. 9 Editor: author behavior … … … … … … … Runtime: observe behavior

10. Outline  Requirements for Alternatives of Programmed Interactions  Juxtapose for Desktop Interactions: System & Evaluation  Alternatives off the Desktop  Related & Future Work 10

11. 3 Requirements for Programmed Interactions  Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 11 (based on 3 interviews & code inspection)

12.  Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 12 Processing code brought in by an interviewee 3 Requirements for Programmed Interactions

13.  Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 13 3 Requirements for Programmed Interactions

14. Juxtapose: Source alternatives… 14

15. Juxtapose: Source alternatives… 15

16. … are executed in parallel, 16

17. and tuned through an generated UI. 17

18. Parallel Editing 18 Juxtapose editor for ActionScript 2. Generates Flash files.

19. Parallel Editing 19

20. Parallel Editing Linked Editing: Toomim 2004 20

21. Parallel Editing Linked Editing: Toomim 2004 21

22. Example: Rethinking the Progress Bar 22 Harrison et al., UIST 2007 Progress profiles Progress bar test application

23. 23

24. Parallel Input: Event Echoing 24

25. Implementing Parallel Editing: The Cursor Correspondence Problem 25 Straightforward solution: Longest common subsequence algorithm diff(A,B,…,N) Run during idle times Doc. A Doc. BDoc. C

26. Limits of Parallel Editing & Execution  Level of granularity for editing: Extension to multi-file projects is not trivial.  Can the user make sense of behaviors running in parallel? 26

27. Limits of Event Echoing 27 Yes No Delete this file? Ok Delete this file? Yes No

28. Parameter Tuning 28

29. Example: Tuning Phosphor 29 Baudisch, UIST 2006

30. Example: Tuning Phosphor 30 Baudisch, UIST 2006

31. Generating Tuning Interfaces Juxtapose User Library User’s Application: //… Number A = 10; Boolean B = true; User’s Application: //… Number A = 10; Boolean B = true; User Process Juxtapose Process Inspect Send variable info XB A 31

32. Generating Tuning Interfaces Juxtapose User Library User’s Application: //… Number A = 10; Boolean B = true; User’s Application: //… Number A = 10; Boolean B = true; User Process Juxtapose Process Update Send tuning message XB A 32

33. What parameter ranges should be chosen? 0.0-1.0? 0-100? 0-255? Number.minValue – Number.maxValue? 33 Any nontrivial program property is undecidable.

34. Take a guess, then give control to the user. 34 At authoring time: source annotations

35. Limits of Tuning  Which variables matter? Juxtapose extracts only globals 35

36. Limits of Tuning  Which variables matter? Juxtapose extracts only globals  Are variables read again? Juxtapose uses callbacks 36 01 var tunable = 5; //@RANGE 0..100 03 var callback= function(varName,oldVal,newVal){ 04 redraw(); 05 return newVal; 06 } 07 this.watch(′tunable′,callback);

37. Understanding User Strategies & Measuring Benefits  Questions:  How does Juxtapose fit into interaction prototyping practice?  Can we quantify the benefits?  Method:  Lab evaluation, N=18, students  75-90 minute sessions with design tasks & survey 37

38. 38 Mapping Task Given ActionScript code that loads a multilayered map, develop navigation options for a handheld GPS prototype for pedestrians and bicyclists.

39. Alternatives for Map Navigation & Rendering 39

40. 40 Participants’ Strategies  Linked vs. unlinked editing with multiple alternatives applied by all participants.  Alternatives used both for different scenarios, as well as “scratch space”  Snapshots important to save state

41. 41 Suggested improvements  Automate code restructuring for callbacks  Introduce direct manipulation for parameters in user’s application (when possible)

42. 42 Tree Matching Task Search in 4D parameter space. Given recursive drawing code for this: Produce these:

43. 43

44. 44

45. 45 p<0.01 p~0.01 not significant

46. Outline  Requirements for Alternatives of Programmed Interactions  Juxtapose for Desktop Interactions: System & Evaluation  Alternatives off the Desktop  Related & Future Work 46

47. Juxtapose Mobile 47 Alternative 1 Alternative 2 Alternative 3

48. 48

49. Juxtapose Runtime on PC Juxtapose Mobile Juxtapose Wrapper User’s Flash Application 49 Juxtapose Wrapper User’s Flash Application TCP/IP var 1 Alt. 1 Alt. 2 var 2

50. Juxtapose for Microcontrollers 50 Arduino: 8bit Atmel AVR RISC chip programmed in C with gcc

51. 51 How might one implement variable tuning in a language without reflection?

52. Parse code and build symbol table Var. NameTypePointer “dly”int&dly “blink”boolean&blink int dly = 25; boolean blink=false; void setup() {...} //… 52

53. Emit table into code & compile void initVarTable(void) { varTable[1].varName = PSTR("dly"); varTable[1].varType = VAR_TYPE_INT; varTable[1].varPtr = &dly; varTable[2].varName = PSTR("blink"); varTable[2].varType = VAR_TYPE_BOOLEAN; varTable[2].varPtr = &blink; } // plus a bunch of communication code... Auto-generated table 53

54. At Runtime Serial message: tune “dly” value 100 *(varTable[“dly”].varPtr) = 100; X dlyblink 54

55. So What’s Different? Target PlatformHow are alternatives executed? Why? How do users interact with alternatives? In parallelSequentially or in parallel In parallel, because commodity hardware is (relatively) cheap Sequentially (parallelism is possible, but less appealing) Sequentially, because custom hardware is expensive to build Sequentially 55

56. What’s Important? A structured approach to alternatives eliminates cruft and enables more exploration. 56

57. What’s Important? A structured approach to alternatives eliminates cruft and enables more exploration. Tool support requires connecting authoring and execution environments. 3 Techniques:  Linked editing to manage code alternatives  Execute set of programs side-by-side  Auto-generate tuning interface 57

58. Related Work 58 Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 (only first authors listed) Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 JPie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 Alternatives in Other DomainsAugmented Development Tools

59. Related Work 59 Language-level support Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 JPie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 Augmented Development Tools

60. Related Work 60 Alternatives are generated automatically from formal specification Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 Alternatives in Other Domains

61. Related Work 61 Exploit spatial syntax Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 Alternatives in Other Domains

62. Current Work: Revisions for Interaction Design 62 Revisions & comments for text in Microsoft Word Revisions & comments for interactions in d.tools

63. stanford hci groupoct 21, 2008 http://hci.stanford.edu Björn Hartmann Loren Yu, Abel Allison, Yeonsoo Yang Scott R. Klemmer Supported by: NSF grant IIS- 0745320 Equipment from Intel & Nokia SAP Stanford Graduate Fellowship

64. 64 At runtime: editable min/max values Take a guess, then give control to the user.

65. Implementing Parallel Editing: Incremental Structure Tracking 65 Doc. A User inserts text Empty block created Blocks sub- divided User deletes text Doc. B Doc. A Doc. B Doc. A Doc. B Doc. A Doc. B INSERTDELETE

66. Future Work Graphics + Code? 66

67. 67

68. 68

69. 69

70. 70

71. Tangible Control 71

72. Switching alternatives Load Alternative2.hex Alternative 2 Bootloader 72

73. 73 p<0.001 (paired two-tailed Student’s t)

74. Buxton, Sketching User Experiences

75. How can interaction design tools support creation and management of user interface alternatives? 75

76. Juxtapose Runtime on PC Juxtapose Mobile Juxtapose Wrapper User’s Flash Application Open socket, Send variable info 76 Juxtapose Wrapper User’s Flash Application TCP/IP Open socket, send variable info var 1 Alt. 1 Alt. 2 var 2

77. Juxtapose Mobile Juxtapose Wrapper User’s Flash Application Tuning messages For Alternative 2 77 Juxtapose Wrapper User’s Flash Application Tuning messages For Alternative 1 TCP/IP Juxtapose Runtime on PC var 1 Alt. 1 Alt. 2 var 2

78. Adding Tuning to the Arduino IDE 78

79. Tuning without Reflection Parse source to extract global variable declarations Emit variable table + wrapper code into source Compile modified source 79

80. Related Work 80 Design Galleries, Marks, SIGGRPAH 97 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 TEAM STORM, Hailpern, CHI 07 Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 Jpie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 (only first authors listed) Mixed initiative

81. Redrawn from Buxton, Sketching User Experiences Generate Options, Select. Repeat.

82. Limits of Parallel Editing  Cognitive challenge: Changes to alternatives are not visible during editing  Level of granularity for editing: Extension to multi-file projects not trivial 82

83. Beyond the Desktop 83