Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999

2 Purpose l Allow nonprogrammers to build applications with original, dynamic behavior n Use programming-by-demonstration n No written language at all - programs created solely through demonstration n Implement in system called Gamut

3 State of the Art l Full-fledged languages and libraries: n Visual C++, Java l Simplified languages with syntax- supporting editor: n Authorware, Agentsheets l Tools with macro languages: n Director / LINGO, Excel / Visual Basic

4 Outline l Introduction l Interaction Techniques l Video l Inferencing l User Study l Conclusion

5 Domain l “Board Games” n Has background “board” n “Pieces” above board show state n Pieces follow rule-like behavior n Autonomous behavior controlled by timers l Examples n Tic-Tac-Toe, Pacman, Reader Rabbit

6 Example: Pacman

7 Innovations in Gamut l Interaction Techniques n Allows developers to specify important details l Inferencing n Infer conditions from sources unaffected by the behavior n Infer chains of expressions

8 Inferencing - Conditions l Relationship can be formed between arbitrary objects –Relating a toggle switch to decide which way to move Run Away ?

9 GO Expression Chain l Expressions form chain of relationships –In Monopoly, the position to move a piece is, “the square which is the sum of the dice’s number of squares from the original position” Red’s Turn

10 Interaction Techniques l Nudges n Do Something, Stop That n Hint highlighting l Guide Objects l Temporal Ghosts l Cards and Decks

11 Nudges

12 Negative Examples l Standard technique for demonstrating examples: Macro Recorder n Requires special controls to make a negative example l Negative examples in Gamut provided by Stop That n No special set-up required

13 Guides, Ghosts, and Hints

14 Cards and Decks

15 Video

16 Stages of Inferencing Translation Matching Actions Code Generation

17 Translation l Read events from undo history list l Convert events to actions l Eliminate redundant and inconsequential events

18 Matching Actions l Matches actions in trace to original behavior –Unordered –Typically not many (three or four) l Considerations –Degree to which actions match –Priority of actions in original behavior –Actions which create new objects

19 Recursive Difference l Propagates changes to descriptions l Each description contains heuristics for comparing values l Builds an And/Or tree result

20 Code Generation l Resolves each value difference by creating a new description n Search for previously defined description n Use heuristics to generate description from highlighted objects n Assemble values into a table and use Decision Tree learning

21 Decision Trees l Created at places where Gamut chooses between different behaviors l Generate attributes using heuristics and hint highlighting l Build table of examples that relate behaviors to attributes Is Eaten Run Home Is Blue Chase Pacman Run Away YN YN Condition For Pacman Monster

22 User Study l Goal n Determine if nonprogrammers can use Gamut n Perform an informal study l Participants n Nonprogrammers, CMU community l Location n HCI Institute’s usability lab

23 Tasks Safari G-bert Pawn Race

24 Expertise with Techniques

25 Problems l Reluctance to create guide objects n Needed explicit instructions in tasks n Once told, participants would create appropriate objects l Highlighting inappropriate objects n Some objects seem too obvious to highlight, like ghosts

26 Future Work l Apply to other domains l Feedback for just inferred relationships l Display for language l Higher-level inferencing

27 Conclusion l Builds entire applications without a written programming language l Infers a broader range of behavior than prior systems l Is usable by nonprogrammers