1. Lecturing with Digital Ink Richard Anderson University of Washington

2. Lessons learned from the Classroom Presenter project Classroom Pedagogy Teaching with ink HCI Ink based presentation Multimedia Analysis of lecture artifacts

3. Classroom Presenter Integration of slides and digital ink using Tablet PC Key ideas: Ink overlay on images Distributed application Many other systems also support ink and slides

4. Classroom Presenter as a distributed application Designed as distributed application for distance learning Enables many scenarios Mobility Walking and talking Sharing materials with students Note taking Classroom interaction Student submissions

5. Deployments Estimated use in at least 100 courses Wide use inside of computer science Push for adoption outside of CS Lecture archives from UW Professional Master’s Program Several hundred hours of recorded audio, video, and ink.

6. Distance Learning Classes

7. “Typical ink usage”

9. Planning for ink usage

10. Ink use in presentation Cognitive load Only limited attention available for computer while lecturing Linkage with speech Close tie between ink and speech

11. Cognitive load Limited feature use Even color change unusual User interface must be simple (and robust) Cannot give feedback to user Many actions appear to minimize mental effort Color change only for contrast Reliance on screen erase

12. Understanding Attentional Marks Properties Brief, simple markings Occur with speech Augment meaning of speech Ad hoc form Is there a linguistic context in which to understand these marks?

13. Spontaneous Hand Gestures Spontaneous Hand gestures [McNeill]: are synchronous w/speech are co-expressive w/speech lack standard of form Attentional marks share these properties.

14. Gesture Types: Iconic

15. Gesture Types: Deictic & Cohesive

16. Analysis of digital ink Understand ink usage Motivation: inform development of ink based applications Archiving Search, Summarization, Transcription Lecture based Improved rendering, note taking, accessibility

17. Ink classification Textual Diagrammatic Attentional % of strokes% of episodes BCB+CBC Attentional495351777476 Diagram978888 Writing413840141615 Other121222 Coding of six hours of lecture

18. Goals Understand usage “in the wild” Cannot expect lecturers to modify behavior Determine opportunities for automatic analysis Identify challenges

19. Methodology Study of recorded classes Best data set: Professional Master’s Program Distance courses Audio, Video, Ink archives HCI, Compilers, Programming Languages, AI, Transaction Processing

20. Attentional ink Problem – content matching Identify slide content referred to by ink Study Implement basic algorithms to match attention marks to slide content Compare results with human coders

21. Attentional ink Determine the lecturer’s intent: Determine level to parse the content

22. Attentional ink Challenges Recognition of attentional ink on text Difficult example:

23. Handwriting How well does handwriting recognition work on “typical” instructor writing? Domain has many challenges

24. Recognition Study Studied isolated words/phrases written on slides Removed non-textual ink Fed through the Microsoft Handwriting Recognizer No training

25. Recognition Examples The Good: The Bad: The Ugly:

26. Handwriting Reco Results ExactAlternateCloseNone Prof. A 16 (88%)1 (6%)0 (0%)1 (6%) Prof. B 146 (59%)26 (10%)6 (2%)71 (29%) Prof. C 18 (42%)5 (11%)1 (3%)19 (44%) Prof. D 262 (61%)45 (11%)9 (2%)111 (26%) Prof. E 408 (79%)46 (9%)2 <(1%)58 (11%) Total 850 (68%)123 (10%)18 (1%)260 (21%)

27. Joint Writing and Speech Recognition Can we use handwriting recognition with speech recognition together to improve accuracy? Co-expression of ink and speech Are written words spoken as well? Can speech disambiguate handwriting? Can handwriting disambiguate speech?

28. Examples Difficult for Speech and Ink Recognition Difficult Written Abbreviations Speech/Ink Used to Disambiguate Ink/Speech

29. Experiment Examined instances of isolated word writing Selected word writing episodes at random but uniformly from the various instructors Generated transcripts manually from the audio Checked whether the instructor spoke the exact word written Measured the time between the written and spoken word

30. Speech/Text Co-occurrence Results

31. Activity Recognition Identifying slide corrections

32. Example Results

33. Diagrammatic ink How do instructors use diagrams Basic legibility Observed behaviors Diagram phasing Locality of expression

35. Typical diagram Basic, irregular shapes Difficult labels Attentional ink

36. More examples

37. Zipf diagram

38. Stroke order

39. Diagram phasing

40. More phasing

41. Top arrows: “Not there” Separate wins indicated together Locality in diagrams

42. Summary Pedagogy with ink How is ink used in conjunction with content and speech to express information Presentation with ink Low attention task Analysis of ink usage Extracting meaning from archived lectures

43. Resources cs.washington.edu/education/dl/presenter/ Software Downloads Papers Contact info Richard Anderson, anderson@cs.washington.eduanderson@cs.washington.edu Ruth Anderson, ruth@cs.virginia.eduruth@cs.virginia.edu Craig Prince, cmprince@cs.washington.educmprince@cs.washington.edu