1. A Trace-based Framework for Analyzing and Synthesizing Educational Progressions Erik Andersen Sumit Gulwani Zoran Popović

2. Procedural Learning

3. Software Learnability Dong et al. CHI 2012

5. Flow Skill Challenge Anxiety Boredom M. Csikszentmihalyi

6. Zone of Proximal Development can do now can do with guidance can’t do yet L. Vygotsky

8. Procedural Traces What are traces? What can we do with traces?

10. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES...

11. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S

12. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L

13. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L, H

14. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L, H, DS

15. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L, H, DS, DG

16. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L, H, DS, DG, DDS

17. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S L, H, DS, DG, DDS, DDG…

18. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S Compare 313 to 324.

19. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S Compare 313 to 324.

20. What can we do with traces? Rank their difficulty Analyze and compare progressions Synthesize new progressions

21. What can we do with traces? Rank their difficulty Analyze and compare progressions Synthesize new progressions

22. Partial Ordering Loop executes longer = harder – more digits in number comparison – more columns in subtraction Exceptional conditions = harder – if boolean expression: statement – borrowing in subtraction – carrying in addition

23. n-grams

24. 1-grams the boy threw the ball

25. 2-grams the boy threw the ball

26. nn-grams 1H, L, D, S, G 2DD, DS, DG 3DDS, DDG

27. n-gram based Partial Ordering

28. What can we do with traces? Rank their difficulty Analyze and compare progressions Synthesize new progressions

29. if length(n 1 ) < length(n 2 ): return less L if length(n 1 ) > length(n 2 ): return greater H for i ← 0 to length(n 1 ) – 1: D if n 1 [i]< n 2 [i]: return less S if n 1 [i]> n 2 [i]: return greater G return equal

30. JUMP Math

31. Singapore Math

32. JUMP Math Singapore Math

33. What can we do with traces? Rank their difficulty Analyze and compare progressions Synthesize new progressions

34. if length(n 1 ) < length(n 2 ): return less L if length(n 1 ) > length(n 2 ): return greater H for i ← 0 to length(n 1 ) – 1: D if n 1 [i]< n 2 [i]: return less S if n 1 [i]> n 2 [i]: return greater G return equal DDS 11 < 12

35. ProblemTimeUnique Traces Addition64 min1433 Fraction Computation16 min72 Fraction Reduction2 min29 Integer Comparison2 min26 Subtraction2 min157 Prime Factorization2 min70

36. DragonBox

39. Validating Our Partial Ordering 20 participants 20 level pairs Users ranked pairs: easier, harder, the same Calculated Root-Mean-Squared-Error – our ordering: 0.35 – random baseline: 1.00

40. Summary new representation of problem difficulty allows us to: – rank problem difficulty – analyze and compare progressions – generate an entire progression pragmatic way to make content fit theory

41. Acknowledgments Help: David Yamanoha and Ryoko Nozawa Funding: Bill and Melinda Gates Foundation, Office of Naval Research, Hewlett Foundation, Adobe, and Intel Contact me: eland@cs.washington.edu

44. From here automate other things – tutorials – diagnosis of student errors

45. Flow Skill Challenge Anxiety Boredom

46. Implications Games Education

47. “Games are a series of interesting decisions.” – Sid Meier

48. Artifacts

49. Artifacts Progressions

50. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S Compare 32 to 314.

51. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S DDDDG 1112 > 1111

54. ProblemBookProblemsTraces% gen AdditionSS:M5841788% SMS5764864% Fraction Addition & Subtraction SS:M6613100% SMS24815100% JUMP1311195% Fraction Reduction SS:M6613100% JUMP5611100% Integer Comparison SS:M246100% JUMP8812100%

55. HCI education software engineering video games

56. Promise new representation of problem difficulty can analyze problem sets generate an entire progression

57. Promise new representation of problem difficulty can analyze problem sets generate an entire progression

58. if length(n 1 ) < length(n 2 ): return less L if length(n 1 ) > length(n 2 ): return greater H for i ← 0 to length(n 1 ) – 1: D if n 1 [i]< n 2 [i]: return less S if n 1 [i]> n 2 [i]: return greater G return equal

59. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES...

60. same # of digits? NO greater # of digits? NO report “less” YES report “greater” YES first digit same?first digit greater? NO report “less” YES report “greater” NO second digit same?second digit greater? NO report “less” YES report “greater” NO YES... D D S L H D G G S DDS 11 < 12

61. DragonBox Level Solver action 1 action 2 action 3... AAB AAAAB ABC

62. n-gram based Partial Ordering the boy passed the ball – unigrams: the, boy, passed, ball – bigrams: the boy, boy passed, passed the, the ball – trigrams: the boy passed, boy passed the, passed the ball

63. n-gram based Partial Ordering Integer Comparison – unigrams: H, L, D, G, S – bigrams: DD, DG, DS – trigrams: DDG, DDS

64. Promise new representation of problem difficulty can analyze problem sets generate an entire progression

65. Promise new representation of problem difficulty can analyze problem sets generate an entire progression

67. Fraction Addition and Subtraction Integer Addition