Using Handhelds to Enhance Classrooms and to Help the Handicapped Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University Brad A. Myers Carnegie Mellon University SUNY Technology Conference 2002

2 Brad Myers Overview in Previous Session l Pebbles project: l How handhelds can be used at the same time as other devices l Many domains, applications and research topics l Pebbles project: l How handhelds can be used at the same time as other devices l Many domains, applications and research topics

SUNY Technology Conference Brad Myers Overview of This Talk l Pebbles applications for classrooms l Slideshow Commander for the presenter l Or RemoteCommander or Shortcutter l Using handhelds for real-time “Concept Tests” l Pebbles applications for the Handicapped l Handhelds as an assistive technology for people with Muscular Dystrophy and related disorders l Pebbles applications for classrooms l Slideshow Commander for the presenter l Or RemoteCommander or Shortcutter l Using handhelds for real-time “Concept Tests” l Pebbles applications for the Handicapped l Handhelds as an assistive technology for people with Muscular Dystrophy and related disorders

Presenter Use l Slideshow Commander for the Presenter

SUNY Technology Conference Brad Myers SlideShow Commander l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time

SUNY Technology Conference Brad Myers Studies of Presentations l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems

SUNY Technology Conference Brad Myers View on Hand-Held: slide l See thumb-nail of current slide l Black and white or color l See thumb-nail of current slide l Black and white or color

SUNY Technology Conference Brad Myers Scribble on Current Slide l Can handwrite and draw on thumbnail picture l Like with overheads l Or check off items as you go l Can handwrite and draw on thumbnail picture l Like with overheads l Or check off items as you go

SUNY Technology Conference Brad Myers New! Tap On-screen Buttons l Mode on handheld switches between scribble and tap l Invoke sounds, animations, etc. l Mode on handheld switches between scribble and tap l Invoke sounds, animations, etc.

SUNY Technology Conference Brad Myers New! Preview forward or backwards l iPaq version only l See what is coming next on the handheld without showing students l Or look back at a previous slide l Jump to that slide without showing ones in between l iPaq version only l See what is coming next on the handheld without showing students l Or look back at a previous slide l Jump to that slide without showing ones in between

SUNY Technology Conference Brad Myers View on Hand-Held: notes l See the notes of the current slide

SUNY Technology Conference Brad Myers View on Hand-Held: titles l See the list of titles

SUNY Technology Conference Brad Myers View on handheld: Time l Palm version only l Multi-function timer l View large l And in corner l Palm version only l Multi-function timer l View large l And in corner

SUNY Technology Conference Brad Myers Control Demonstrations on PocketPC l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume

SUNY Technology Conference Brad Myers Control Demonstrations on Palm l Control PC applications and external devices l Uses Shortcutter l Create panels of controls for any app l Also can control lights and projector l Easy demo and resume show l Control PC applications and external devices l Uses Shortcutter l Create panels of controls for any app l Also can control lights and projector l Easy demo and resume show

SUNY Technology Conference Brad Myers Commercialized l SlideShow Commander commercialized by: l Available from Office Depot, CompUSA, etc. l Palm and PocketPC l (Some demonstrated features not available in commercial version) l SlideShow Commander commercialized by: l Available from Office Depot, CompUSA, etc. l Palm and PocketPC l (Some demonstrated features not available in commercial version) Synergy Solutions, Inc. Synergy Solutions, Inc.

SUNY Technology Conference Brad Myers But what if not using PowerPoint? l Can use RemoteCommander to control any PC application l Can use Shortcutter to send specific keystrokes l One professor uses Shortcutter to control Adobe Acrobat l Still only PC l Can use RemoteCommander to control any PC application l Can use Shortcutter to send specific keystrokes l One professor uses Shortcutter to control Adobe Acrobat l Still only PC

Student Use of Handhelds l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor

SUNY Technology Conference Brad Myers Hardware l HP donated 110 Jornada 680 and computers l Windows CE l Lucent donated Wavelan wireless cards l HP donated 110 Jornada 680 and computers l Windows CE l Lucent donated Wavelan wireless cards

SUNY Technology Conference Brad Myers Context l Collaborator: Prof. David Yaron of Chemistry l Chemistry : Modern Chemistry II l Spring, 2000 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student l Collaborator: Prof. David Yaron of Chemistry l Chemistry : Modern Chemistry II l Spring, 2000 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student

SUNY Technology Conference Brad Myers Wireless Andrew l CMU has almost complete coverage of b support l Wavelan (now ORiNOCO) l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support l CMU has almost complete coverage of b support l Wavelan (now ORiNOCO) l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support

SUNY Technology Conference Brad Myers Testing l Large lectures are a poor way to educate l Work on engaging students l So they pay attention and think about material l “Concept Tests” l Short tests given during lectures l Students discuss, and answer questions again l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall. l Large lectures are a poor way to educate l Work on engaging students l So they pay attention and think about material l “Concept Tests” l Short tests given during lectures l Students discuss, and answer questions again l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall.

SUNY Technology Conference Brad Myers Concept Tests l Students have to think about answers l Instructor can determine who is participating l Can determine what students understand l Computerize: l Use technology to provide capabilities not otherwise available l Rather than just a new way to do old things l Students have to think about answers l Instructor can determine who is participating l Can determine what students understand l Computerize: l Use technology to provide capabilities not otherwise available l Rather than just a new way to do old things

SUNY Technology Conference Brad Myers Implementation l Use web-based tests for students l Web-based instructor’s display l Hand-helds and instructor’s computer use standard IE browser l Server provides database of tests l Currently only multiple-choice l Server also stores students’ answers l Tests represented as XML l Use web-based tests for students l Web-based instructor’s display l Hand-helds and instructor’s computer use standard IE browser l Server provides database of tests l Currently only multiple-choice l Server also stores students’ answers l Tests represented as XML

SUNY Technology Conference Brad Myers Pebbles Tests l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers

SUNY Technology Conference Brad Myers Concept Test l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board and didn’t author in html l Used for lectures and also during in-class chemistry demonstrations and experiments l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board and didn’t author in html l Used for lectures and also during in-class chemistry demonstrations and experiments

SUNY Technology Conference Brad Myers Instructor’s View l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students

SUNY Technology Conference Brad Myers For Instructor l Faculty can get data from prior tests l Enables grading, trend analysis l By student name, or anonymous l Faculty can get data from prior tests l Enables grading, trend analysis l By student name, or anonymous 

SUNY Technology Conference Brad Myers Results l Unfortunately, no data about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. l Unfortunately, no data about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. Raise HandsHold up cardsHandhelds 25%4%63%

SUNY Technology Conference Brad Myers Problems l Connection to network worked? l How often needed to reset? l Connection to network worked? l How often needed to reset? AlwaysSometimesNever 15%67%4% Never1-2 times3-5 times> 5 tlmes 4%23%19%42%

SUNY Technology Conference Brad Myers Problems, cont. l Biggest drawback: l Keyboard was: l Screen size was: Crashes too often 27% Other… 23% Not powerful enough 15% Too big 12% No major drawbacks 10% AdequateToo smallPrefer no keyboard 15%67%4% AdequateToo smallWant smaller device 15%67%4%

SUNY Technology Conference Brad Myers Future l Hope for further study of impact of handhelds l This software is available for you to download off the main Pebbles web page: l webserversoftware l Written in Perl5 l Hope for further study of impact of handhelds l This software is available for you to download off the main Pebbles web page: l webserversoftware l Written in Perl5

Handhelds for the Handicapped l Handhelds as Assistive Technology l As an alternative input device for a computer l Handhelds as Assistive Technology l As an alternative input device for a computer

SUNY Technology Conference Brad Myers History l Drew Rossman found RemoteCommander on the web l Found it helped his daughter who has a form of Muscular Dystrophy known as Spinal Muscular Atrophy, Type II. l Wrote articles and created web page to help promote its use l We adapted the software to try to improve performance l Drew Rossman found RemoteCommander on the web l Found it helped his daughter who has a form of Muscular Dystrophy known as Spinal Muscular Atrophy, Type II. l Wrote articles and created web page to help promote its use l We adapted the software to try to improve performance

SUNY Technology Conference Brad Myers Muscular Dystrophy l About 250,000 people in the United States have Muscular Dystrophy (MD) l Use computers for recreation, education, etc. l Often home-bound, so computers may be even-more important l Increased susceptibility to infections l Web provides opportunities for employment l People with MD often have fine-motor control but lose gross motor control l Difficulty with mouse and keyboard, but stylus OK l May tire easily, so having choice of different input devices can be useful l About 250,000 people in the United States have Muscular Dystrophy (MD) l Use computers for recreation, education, etc. l Often home-bound, so computers may be even-more important l Increased susceptibility to infections l Web provides opportunities for employment l People with MD often have fine-motor control but lose gross motor control l Difficulty with mouse and keyboard, but stylus OK l May tire easily, so having choice of different input devices can be useful

SUNY Technology Conference Brad Myers Alternative Input Devices l Commercial alternative input devices can be expensive and difficult to use l Must move from keyboard device to trackball device l So use inexpensive Palm instead l Commercial alternative input devices can be expensive and difficult to use l Must move from keyboard device to trackball device l So use inexpensive Palm instead

SUNY Technology Conference Brad Myers Adaptations to Remote Commander l Made on-screen buttons for left, right mouse buttons and keyboard modifiers l Physical Palm buttons too difficult to press l Made on-screen buttons for left, right mouse buttons and keyboard modifiers l Physical Palm buttons too difficult to press

SUNY Technology Conference Brad Myers Adaptations, cont. l Turn off key repeat l So not repeating by accident l Faster Acceleration l Smaller movements on Palm l More flexible tapping l To get mouse button down l Long time before power-off l Palm power button too hard to press l Turn off key repeat l So not repeating by accident l Faster Acceleration l Smaller movements on Palm l More flexible tapping l To get mouse button down l Long time before power-off l Palm power button too hard to press

SUNY Technology Conference Brad Myers Adaptations, cont. l Various Keyboard Layouts l Bigger buttons l Alphabetic layout l Most common keys grouped together to minimize travel distance l Various Keyboard Layouts l Bigger buttons l Alphabetic layout l Most common keys grouped together to minimize travel distance

SUNY Technology Conference Brad Myers New! Word Prediction l Predicts word from first letters l Predicts next word from previous word l Layout to minimize distance from keys l Most likely choice in center and highlighted l Available with soft keyboard and Graffiti l Predicts word from first letters l Predicts next word from previous word l Layout to minimize distance from keys l Most likely choice in center and highlighted l Available with soft keyboard and Graffiti

SUNY Technology Conference Brad Myers Case Studies l Observed four people with MD using Remote Commander l Measured typing and mouse speed with PC-based tests l Observed four people with MD using Remote Commander l Measured typing and mouse speed with PC-based tests

SUNY Technology Conference Brad Myers Observations l Design changes: l Many of the design decisions already mentioned l Needed lighter & longer stylus l Needed better lighting l Cradle not sufficiently stable: needed long cord; l Positives: l Tiny keyboard requiring small movements l Keyboard and mouse in one place l Ability to control PC from across room l Ability to work with regular keyboard and mouse, for shared computers l Problems: l Battery life is a real barrier l Still much slower than regular keyboard l Design changes: l Many of the design decisions already mentioned l Needed lighter & longer stylus l Needed better lighting l Cradle not sufficiently stable: needed long cord; l Positives: l Tiny keyboard requiring small movements l Keyboard and mouse in one place l Ability to control PC from across room l Ability to work with regular keyboard and mouse, for shared computers l Problems: l Battery life is a real barrier l Still much slower than regular keyboard

SUNY Technology Conference Brad Myers Future work l Handhelds as assistive technologies in other areas l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.? l Handhelds to help with other disabilities l For the visually impaired? l SlideShow Commander l Auditory interface to PC apps, and the environment? l Ideas welcome! l Handhelds as assistive technologies in other areas l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.? l Handhelds to help with other disabilities l For the visually impaired? l SlideShow Commander l Auditory interface to PC apps, and the environment? l Ideas welcome!

SUNY Technology Conference Brad Myers Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

Using Handhelds to Enhance Classrooms and to Help the Handicapped Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University Brad A. Myers Carnegie Mellon University SUNY Technology Conference 2002