1. Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999

2. 2 Purpose l Allow nonprogrammers to build applications with original, dynamic behavior n Use programming-by-demonstration n No written language at all - programs created solely through demonstration n Implement in system called Gamut

3. 3 State of the Art l Full-fledged languages and libraries: n Visual C++, Java l Simplified languages with syntax- supporting editor: n Authorware, Agentsheets l Tools with macro languages: n Director / LINGO, Excel / Visual Basic

4. 4 Outline l Introduction l Interaction Techniques l Video l Inferencing l User Study l Conclusion

5. 5 Domain l “Board Games” n Has background “board” n “Pieces” above board show state n Pieces follow rule-like behavior n Autonomous behavior controlled by timers l Examples n Tic-Tac-Toe, Pacman, Reader Rabbit

6. 6 Example: Pacman

7. 7 Innovations in Gamut l Interaction Techniques n Allows developers to specify important details l Inferencing n Infer conditions from sources unaffected by the behavior n Infer chains of expressions

8. 8 Inferencing - Conditions l Relationship can be formed between arbitrary objects –Relating a toggle switch to decide which way to move Run Away ?

9. 9 GO Expression Chain l Expressions form chain of relationships –In Monopoly, the position to move a piece is, “the square which is the sum of the dice’s number of squares from the original position” Red’s Turn

10. 10 Interaction Techniques l Nudges n Do Something, Stop That n Hint highlighting l Guide Objects l Temporal Ghosts l Cards and Decks

11. 11 Nudges

12. 12 Negative Examples l Standard technique for demonstrating examples: Macro Recorder n Requires special controls to make a negative example l Negative examples in Gamut provided by Stop That n No special set-up required

13. 13 Guides, Ghosts, and Hints

14. 14 Cards and Decks

15. 15 Video

16. 16 Stages of Inferencing Translation Matching Actions Code Generation

17. 17 Translation l Read events from undo history list l Convert events to actions l Eliminate redundant and inconsequential events

18. 18 Matching Actions l Matches actions in trace to original behavior –Unordered –Typically not many (three or four) l Considerations –Degree to which actions match –Priority of actions in original behavior –Actions which create new objects

19. 19 Recursive Difference l Propagates changes to descriptions l Each description contains heuristics for comparing values l Builds an And/Or tree result

20. 20 Code Generation l Resolves each value difference by creating a new description n Search for previously defined description n Use heuristics to generate description from highlighted objects n Assemble values into a table and use Decision Tree learning

21. 21 Decision Trees l Created at places where Gamut chooses between different behaviors l Generate attributes using heuristics and hint highlighting l Build table of examples that relate behaviors to attributes Is Eaten Run Home Is Blue Chase Pacman Run Away YN YN Condition For Pacman Monster

22. 22 User Study l Goal n Determine if nonprogrammers can use Gamut n Perform an informal study l Participants n Nonprogrammers, CMU community l Location n HCI Institute’s usability lab

23. 23 Tasks Safari G-bert Pawn Race

24. 24 Expertise with Techniques

25. 25 Problems l Reluctance to create guide objects n Needed explicit instructions in tasks n Once told, participants would create appropriate objects l Highlighting inappropriate objects n Some objects seem too obvious to highlight, like ghosts

26. 26 Future Work l Apply to other domains l Feedback for just inferred relationships l Display for language l Higher-level inferencing

27. 27 Conclusion l Builds entire applications without a written programming language l Infers a broader range of behavior than prior systems l Is usable by nonprogrammers