1. Using Handhelds to Enhance Classrooms and to Help the Handicapped Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles SUNY Technology Conference 2002

2. 2 Brad Myers Overview in Previous Session l Pebbles project: l How handhelds can be used at the same time as other devices l Many domains, applications and research topics l Pebbles project: l How handhelds can be used at the same time as other devices l Many domains, applications and research topics

3. SUNY Technology Conference 2002 3 Brad Myers Overview of This Talk l Pebbles applications for classrooms l Slideshow Commander for the presenter l Or RemoteCommander or Shortcutter l Using handhelds for real-time “Concept Tests” l Pebbles applications for the Handicapped l Handhelds as an assistive technology for people with Muscular Dystrophy and related disorders l Pebbles applications for classrooms l Slideshow Commander for the presenter l Or RemoteCommander or Shortcutter l Using handhelds for real-time “Concept Tests” l Pebbles applications for the Handicapped l Handhelds as an assistive technology for people with Muscular Dystrophy and related disorders

4. Presenter Use l Slideshow Commander for the Presenter

5. SUNY Technology Conference 2002 5 Brad Myers SlideShow Commander l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time

6. SUNY Technology Conference 2002 6 Brad Myers Studies of Presentations l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems

7. SUNY Technology Conference 2002 7 Brad Myers View on Hand-Held: slide l See thumb-nail of current slide l Black and white or color l See thumb-nail of current slide l Black and white or color

8. SUNY Technology Conference 2002 8 Brad Myers Scribble on Current Slide l Can handwrite and draw on thumbnail picture l Like with overheads l Or check off items as you go l Can handwrite and draw on thumbnail picture l Like with overheads l Or check off items as you go

9. SUNY Technology Conference 2002 9 Brad Myers New! Tap On-screen Buttons l Mode on handheld switches between scribble and tap l Invoke sounds, animations, etc. l Mode on handheld switches between scribble and tap l Invoke sounds, animations, etc.

10. SUNY Technology Conference 2002 10 Brad Myers New! Preview forward or backwards l iPaq version only l See what is coming next on the handheld without showing students l Or look back at a previous slide l Jump to that slide without showing ones in between l iPaq version only l See what is coming next on the handheld without showing students l Or look back at a previous slide l Jump to that slide without showing ones in between

11. SUNY Technology Conference 2002 11 Brad Myers View on Hand-Held: notes l See the notes of the current slide

12. SUNY Technology Conference 2002 12 Brad Myers View on Hand-Held: titles l See the list of titles

13. SUNY Technology Conference 2002 13 Brad Myers View on handheld: Time l Palm version only l Multi-function timer l View large l And in corner l Palm version only l Multi-function timer l View large l And in corner

14. SUNY Technology Conference 2002 14 Brad Myers Control Demonstrations on PocketPC l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume

15. SUNY Technology Conference 2002 15 Brad Myers Control Demonstrations on Palm l Control PC applications and external devices l Uses Shortcutter l Create panels of controls for any app l Also can control lights and projector l Easy demo and resume show l Control PC applications and external devices l Uses Shortcutter l Create panels of controls for any app l Also can control lights and projector l Easy demo and resume show

16. SUNY Technology Conference 2002 16 Brad Myers Commercialized l SlideShow Commander commercialized by: l Available from Office Depot, CompUSA, etc. l Palm and PocketPC l (Some demonstrated features not available in commercial version) l SlideShow Commander commercialized by: l Available from Office Depot, CompUSA, etc. l Palm and PocketPC l (Some demonstrated features not available in commercial version) Synergy Solutions, Inc. http://www.synsolutions.com/ Synergy Solutions, Inc. http://www.synsolutions.com/ http://www.slideshowcommander.com/

17. SUNY Technology Conference 2002 17 Brad Myers But what if not using PowerPoint? l Can use RemoteCommander to control any PC application l Can use Shortcutter to send specific keystrokes l One professor uses Shortcutter to control Adobe Acrobat l Still only PC l Can use RemoteCommander to control any PC application l Can use Shortcutter to send specific keystrokes l One professor uses Shortcutter to control Adobe Acrobat l Still only PC

18. Student Use of Handhelds l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor

19. SUNY Technology Conference 2002 19 Brad Myers Hardware l HP donated 110 Jornada 680 and 100 720 computers l Windows CE l Lucent donated Wavelan wireless cards l HP donated 110 Jornada 680 and 100 720 computers l Windows CE l Lucent donated Wavelan wireless cards

20. SUNY Technology Conference 2002 20 Brad Myers Context l Collaborator: Prof. David Yaron of Chemistry l Chemistry 09-106: Modern Chemistry II l Spring, 2000 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student l Collaborator: Prof. David Yaron of Chemistry l Chemistry 09-106: Modern Chemistry II l Spring, 2000 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student

21. SUNY Technology Conference 2002 21 Brad Myers Wireless Andrew l CMU has almost complete coverage of 802.11b support l Wavelan (now ORiNOCO) l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support l CMU has almost complete coverage of 802.11b support l Wavelan (now ORiNOCO) l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support

22. SUNY Technology Conference 2002 22 Brad Myers Testing l Large lectures are a poor way to educate l Work on engaging students l So they pay attention and think about material l “Concept Tests” l Short tests given during lectures l Students discuss, and answer questions again l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall. l Large lectures are a poor way to educate l Work on engaging students l So they pay attention and think about material l “Concept Tests” l Short tests given during lectures l Students discuss, and answer questions again l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall.

23. SUNY Technology Conference 2002 23 Brad Myers Concept Tests l Students have to think about answers l Instructor can determine who is participating l Can determine what students understand l Computerize: l Use technology to provide capabilities not otherwise available l Rather than just a new way to do old things l Students have to think about answers l Instructor can determine who is participating l Can determine what students understand l Computerize: l Use technology to provide capabilities not otherwise available l Rather than just a new way to do old things

24. SUNY Technology Conference 2002 24 Brad Myers Implementation l Use web-based tests for students l Web-based instructor’s display l Hand-helds and instructor’s computer use standard IE browser l Server provides database of tests l Currently only multiple-choice l Server also stores students’ answers l Tests represented as XML l Use web-based tests for students l Web-based instructor’s display l Hand-helds and instructor’s computer use standard IE browser l Server provides database of tests l Currently only multiple-choice l Server also stores students’ answers l Tests represented as XML

25. SUNY Technology Conference 2002 25 Brad Myers Pebbles Tests l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers

26. SUNY Technology Conference 2002 26 Brad Myers Concept Test l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board and didn’t author in html l Used for lectures and also during in-class chemistry demonstrations and experiments l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board and didn’t author in html l Used for lectures and also during in-class chemistry demonstrations and experiments

27. SUNY Technology Conference 2002 27 Brad Myers Instructor’s View l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students

28. SUNY Technology Conference 2002 28 Brad Myers For Instructor l Faculty can get data from prior tests l Enables grading, trend analysis l By student name, or anonymous l Faculty can get data from prior tests l Enables grading, trend analysis l By student name, or anonymous 

29. SUNY Technology Conference 2002 29 Brad Myers Results l Unfortunately, no data about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. l Unfortunately, no data about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. Raise HandsHold up cardsHandhelds 25%4%63%

30. SUNY Technology Conference 2002 30 Brad Myers Problems l Connection to network worked? l How often needed to reset? l Connection to network worked? l How often needed to reset? AlwaysSometimesNever 15%67%4% Never1-2 times3-5 times> 5 tlmes 4%23%19%42%

31. SUNY Technology Conference 2002 31 Brad Myers Problems, cont. l Biggest drawback: l Keyboard was: l Screen size was: Crashes too often 27% Other… 23% Not powerful enough 15% Too big 12% No major drawbacks 10% AdequateToo smallPrefer no keyboard 15%67%4% AdequateToo smallWant smaller device 15%67%4%

32. SUNY Technology Conference 2002 32 Brad Myers Future l Hope for further study of impact of handhelds l This software is available for you to download off the main Pebbles web page: http://www.pebbles.hcii.cmu.edu/classroom/ l webserversoftware l Written in Perl5 l Hope for further study of impact of handhelds l This software is available for you to download off the main Pebbles web page: http://www.pebbles.hcii.cmu.edu/classroom/ l webserversoftware l Written in Perl5

33. Handhelds for the Handicapped l Handhelds as Assistive Technology l As an alternative input device for a computer l Handhelds as Assistive Technology l As an alternative input device for a computer

34. SUNY Technology Conference 2002 34 Brad Myers History l Drew Rossman found RemoteCommander on the web l Found it helped his daughter who has a form of Muscular Dystrophy known as Spinal Muscular Atrophy, Type II. l Wrote articles and created web page to help promote its use l We adapted the software to try to improve performance l Drew Rossman found RemoteCommander on the web l Found it helped his daughter who has a form of Muscular Dystrophy known as Spinal Muscular Atrophy, Type II. l Wrote articles and created web page to help promote its use l We adapted the software to try to improve performance

35. SUNY Technology Conference 2002 35 Brad Myers Muscular Dystrophy l About 250,000 people in the United States have Muscular Dystrophy (MD) l Use computers for recreation, education, etc. l Often home-bound, so computers may be even-more important l Increased susceptibility to infections l Web provides opportunities for employment l People with MD often have fine-motor control but lose gross motor control l Difficulty with mouse and keyboard, but stylus OK l May tire easily, so having choice of different input devices can be useful l About 250,000 people in the United States have Muscular Dystrophy (MD) l Use computers for recreation, education, etc. l Often home-bound, so computers may be even-more important l Increased susceptibility to infections l Web provides opportunities for employment l People with MD often have fine-motor control but lose gross motor control l Difficulty with mouse and keyboard, but stylus OK l May tire easily, so having choice of different input devices can be useful

36. SUNY Technology Conference 2002 36 Brad Myers Alternative Input Devices l Commercial alternative input devices can be expensive and difficult to use l Must move from keyboard device to trackball device l So use inexpensive Palm instead l Commercial alternative input devices can be expensive and difficult to use l Must move from keyboard device to trackball device l So use inexpensive Palm instead

37. SUNY Technology Conference 2002 37 Brad Myers Adaptations to Remote Commander l Made on-screen buttons for left, right mouse buttons and keyboard modifiers l Physical Palm buttons too difficult to press l Made on-screen buttons for left, right mouse buttons and keyboard modifiers l Physical Palm buttons too difficult to press

38. SUNY Technology Conference 2002 38 Brad Myers Adaptations, cont. l Turn off key repeat l So not repeating by accident l Faster Acceleration l Smaller movements on Palm l More flexible tapping l To get mouse button down l Long time before power-off l Palm power button too hard to press l Turn off key repeat l So not repeating by accident l Faster Acceleration l Smaller movements on Palm l More flexible tapping l To get mouse button down l Long time before power-off l Palm power button too hard to press

39. SUNY Technology Conference 2002 39 Brad Myers Adaptations, cont. l Various Keyboard Layouts l Bigger buttons l Alphabetic layout l Most common keys grouped together to minimize travel distance l Various Keyboard Layouts l Bigger buttons l Alphabetic layout l Most common keys grouped together to minimize travel distance

40. SUNY Technology Conference 2002 40 Brad Myers New! Word Prediction l Predicts word from first letters l Predicts next word from previous word l Layout to minimize distance from keys l Most likely choice in center and highlighted l Available with soft keyboard and Graffiti l Predicts word from first letters l Predicts next word from previous word l Layout to minimize distance from keys l Most likely choice in center and highlighted l Available with soft keyboard and Graffiti

41. SUNY Technology Conference 2002 41 Brad Myers Case Studies l Observed four people with MD using Remote Commander l Measured typing and mouse speed with PC-based tests l Observed four people with MD using Remote Commander l Measured typing and mouse speed with PC-based tests

42. SUNY Technology Conference 2002 42 Brad Myers Observations l Design changes: l Many of the design decisions already mentioned l Needed lighter & longer stylus l Needed better lighting l Cradle not sufficiently stable: needed long cord; l Positives: l Tiny keyboard requiring small movements l Keyboard and mouse in one place l Ability to control PC from across room l Ability to work with regular keyboard and mouse, for shared computers l Problems: l Battery life is a real barrier l Still much slower than regular keyboard l Design changes: l Many of the design decisions already mentioned l Needed lighter & longer stylus l Needed better lighting l Cradle not sufficiently stable: needed long cord; l Positives: l Tiny keyboard requiring small movements l Keyboard and mouse in one place l Ability to control PC from across room l Ability to work with regular keyboard and mouse, for shared computers l Problems: l Battery life is a real barrier l Still much slower than regular keyboard

43. SUNY Technology Conference 2002 43 Brad Myers Future work l Handhelds as assistive technologies in other areas l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.? l Handhelds to help with other disabilities l For the visually impaired? l SlideShow Commander l Auditory interface to PC apps, and the environment? l Ideas welcome! l Handhelds as assistive technologies in other areas l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.? l Handhelds to help with other disabilities l For the visually impaired? l SlideShow Commander l Auditory interface to PC apps, and the environment? l Ideas welcome!

44. SUNY Technology Conference 2002 44 Brad Myers Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

45. Using Handhelds to Enhance Classrooms and to Help the Handicapped Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles SUNY Technology Conference 2002