1. © 2010 Pearson Addison-Wesley. All rights reserved. Addison Wesley is an imprint of Designing the User Interface: Strategies for Effective Human-Computer Interaction Fifth Edition Ben Shneiderman & Catherine Plaisant in collaboration with Maxine S. Cohen and Steven M. Jacobs CHAPTER 8: Interaction Devices

2. 1-2 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts QWERTY layout is the most common modern-day keyboard layout. The name comes from the first six keys appearing on the top. –1870 Christopher Latham Sholes American inventor who invented first typewriter and the QWERTY keyboard still in use today.QWERTY –good mechanical design and a clever placement of the letters that slowed down the users enough that key jamming was infrequent –put frequently used letter pairs far apart, thereby increasing finger travel distances Dvorak layout is a revised keyboard, and ergonomic alternative to the layout commonly found invented in 1936 by the US physician and professor Dr. August Dvorak (1894-1975) after extensive –reduces finger travel distances by at least one order of magnitude –Acceptance has been slow despite the dedicated efforts of some devotees Dvorak Keyboard 3:12Dvorak Keyboard 8-2

3. 1-3 © 2010 Pearson Addison-Wesley. All rights reserved. Keys Current keyboards have been extensively tested –Size –Shape –Required force –Spacing Speed vs. error rates for majority of users Distinctive click gives audio feedback –Why membrane keyboards are slow (Atari 400?) Environment hazards might necessitate Usually speed is not a factor

4. 1-4 © 2010 Pearson Addison-Wesley. All rights reserved. Keys Guidelines Special keys should be denoted State keys (such as caps, etc.) should have easily noted states Special curves or dots for home keys for touch typists Inverted T Cursor movement keys are important (though cross is easier for novices) Auto-repeat feature –Improves performance, but only if repeat is customizable (motor impaired, young, old) Two thinking points: –Why are home keys fastest to type? –Why are certain keys larger? (Enter, Shift, Space bar) This is called Fitt’s Law

5. 1-5 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts 8-5

6. 1-6 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) ABCDE style 1:30 ABCDE style –26 letters of the alphabet laid out in alphabetical order nontypists will find it easier to locate the keys Additional keyboard issues –IBM PC keyboard was widely criticized because of the placement of a few keys backslash key where most typists expect SHIFT key placement of several special characters near the ENTER key –Number pad layout and wrist and hand placement 8-6

7. 1-7 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) Keys –1/2 inch square keys –1/4 inch spacing between keys –slight concave surface –matte finish to reduce glare finger slippage –40- to 125-gram force to activate –3 to 5 millimeters displacement –tactile and audible feedback important –certain keys should be larger (e.g. ENTER, SHIFT, CTRL) –some keys require state indicator, such as lowered position or light indicator (e.g. CAPS LOCK) –key labels should be large, meaningful, permanent –some "home" keys may have additional features, such as deeper cavity or small raised dot, to help user locate their fingers properly (caution - no standard for this) 8-7

8. 1-8 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) Function keys –users must either remember each key's function, identify them from the screen's display, or use a template over the keys in order to identify them properly –can reduce number of keystrokes and errors –meaning of each key can change with each application –placement on keyboard can affect efficient use –lights next to keys used to indicate availability of the function, or on/off status –typically simply labeled F1, F2, etc, though some may also have meaningful labels, such as CUT, COPY, etc. –frequent movement between keyboard home position and mouse or function keys can be disruptive to use –alternative is to use closer keys (e.g. ALT or CTRL) and one letter to indicate special function - Changing Keyboard layouts in Windows Vista/7 8-8

9. 1-9 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) Cursor movement keys –up, down, left, right –some keyboards also provide diagonals –best layout is natural positions –inverted-T positioning allows users to place their middle three fingers in a way that reduces hand and finger movement –cross arrangement better for novices than linear or box –typically include typamatic (auto-repeat) feature –important for form-fillin and direct manipulation –other movements may be performed with other keys, such as TAB, ENTER, HOME, etc. 8-9

10. 1-10 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) Keyboard and keypads for small devices –Wireless or foldable keyboards –Virtual keyboards –Cloth keyboards –Soft keys –Pens and touchscreens 8-10

11. 1-11 © 2010 Pearson Addison-Wesley. All rights reserved. Keypads for small devices PDAs, Cellphones, Game consoles Fold out keyboards Virtual keyboard Cloth keyboards (ElekSen) Haptic feedback? Mobile phones –Combine static keys with dynamic soft keys –Multi-tap a key to get to a character –Study: Predictive techniques greatly improve performance –Ex. LetterWise = 20 wpm vs 15 wpm multitap Draw keyboard on screen and tap w/ pen –Speed: 20 to 30 wpm (Sears ’93) Handwriting recognition (still hard) –Subset: Graffiti2 (uses unistrokes)

12. 1-12 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) 8-12 The popular RIM Blackberry (http://www.blackberry.com) shown here on the left demonstrated that many people could use a reduced-size keyboard on a regular basis; users typically type with one finger or with both thumbs. The Nokia device in the middle shows that non-English-speaking countries may use different keyboard layouts (here, a French AZERTY keyboard). On the right, a larger keyboard uses the longer dimension of the device and can be slid back into the device when not needed (http://www.nokia.com).

13. 1-13 © 2010 Pearson Addison-Wesley. All rights reserved. Keyboard Layouts (cont.) 8-13 Dasher predicts probable characters and words as users make their selections in a continuous two-dimensional stream of choices

14. 1-14 © 2010 Pearson Addison-Wesley. All rights reserved. Other text entry methods 8-14 The virtual keyboard of the Apple iPhone gains precision by allowing finger repositioning and then activates on lift-off

15. 1-15 © 2010 Pearson Addison-Wesley. All rights reserved. Other text entry methods (cont.) 8-15 Another method is to handwrite on a touch sensitive surface, typically with a stylus using Graffiti® on the Palm devices

16. 1-16 © 2010 Pearson Addison-Wesley. All rights reserved. Pointing Devices Pointing devices are applicable in six types of interaction tasks: 1. Select: –user chooses from a set of items. –used for traditional menu selection, identification of a file in a directory, or marking of a part in an automobile design. 2. Position: –user chooses a point in a one-, two-, three-, or higher-dimensional space –used to create a drawing, to place a new window, or to drag a block of text in a figure. 3. Orient: –user chooses a direction in a two-, three-, or higher-dimensional space. –direction may simply rotate a symbol on the screen, indicate a direction of motion for a space ship, or control the operation of a robot arm. 4. Path: –user rapidly performs a series of position and orient operations. –may be realized as a curving line in a drawing program, the instructions for a cloth cutting machine, or the route on a map. 5. Quantify: –user specifies a numeric value. –usually a one-dimensional selection of integer or real values to set parameters, such as the page number in a document, the velocity of a ship, or the amplitude of a sound. 6. Text: –user enters, moves, and edits text in a two-dimensional space. The –pointing device indicates the location of an insertion, deletion, or change. –more elaborate tasks, such as centering; margin setting; font sizes; highlighting, such as boldface or underscore; and page layout. 8-16 New Technology of Computer Mouse New Technology of Computer Mouse 2:09 (Pointing Devices)

17. 1-17 © 2010 Pearson Addison-Wesley. All rights reserved. Pointing Devices 8-17

18. 1-18 © 2010 Pearson Addison-Wesley. All rights reserved. Direct-control pointing devices lightpen –enabled users to point to a spot on a screen and to perform a select, position, or other task –it allows direct control by pointing to a spot on the display –incorporates a button for the user to press when the cursor is resting on the desired spot on the screen –lightpen has three disadvantages: users' hands obscured part of the screen, users had to remove their hands from the keyboard, and users had to pick up the lightpen 8-18

19. 1-19 © 2010 Pearson Addison-Wesley. All rights reserved. Direct-control pointing devices (cont.) Touchscreen - Reality touchscreen University of Groningen Reality touchscreen University of Groningen –allows direct control touches on the screen using a finger –early designs were rightly criticized for causing fatigue, hand- obscuring-the-screen, hand-off-keyboard, imprecise pointing, and the eventual smudging of the display –lift-off strategy enables users to point at a single pixel –the users touch the surface –then see a cursor that they can drag around on the display –when the users are satisfied with the position, they lift their fingers off the display to activate –can produce varied displays to suit the task –are fabricated integrally with display surfaces 8-19

20. 1-20 © 2010 Pearson Addison-Wesley. All rights reserved. Direct-control pointing devices (cont.) Tablet PCs and Mobile Devices: Natural to point on the LCD surface Stylus Keep context in view Pick up & put down stylus Gestures and handwriting recognition 8-20 Microsoft Surface 2Microsoft Surface 2 5:30

21. 1-21 © 2010 Pearson Addison-Wesley. All rights reserved. Direct-control pointing First device – lightpen –Point to a place on screen and press a button –Pros: Easy to understand and use Very fast for some operations (e.g. drawing) –Cons: Hand gets tired fast! Hand and pen blocks view of screen Fragile Evolved into the touchscreen –Pros: Very robust, no moving parts –Cons: Depending on app, accuracy could be an issue 1600x1600 res with acoustic wave –Must be careful about software design for selection (land-on strategy). If you don’t show a cursor of where you are selecting, users get confused –User confidence is improved with a good lift-off strategy

22. 1-22 © 2010 Pearson Addison-Wesley. All rights reserved. Direct-control pointing Primarily for novice users or large user base Case study: Disney World Need to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

23. 1-23 © 2010 Pearson Addison-Wesley. All rights reserved. Indirect-Control Pointing Pros: –Reduces hand-fatigue –Reduces obscuration problems Cons: –Increases cognitive load –Spatial ability comes more into play Mouse –Pros: Familiarity Wide availability Low cost Easy to use Accurate –Cons: Time to grab mouse Desk space Encumbrance (wire), dirt Long motions aren’t easy or obvious (pick up and replace) –Consider, weight, size, style, # of buttons, force feedback

24. 1-24 © 2010 Pearson Addison-Wesley. All rights reserved. Indirect-Control Pointing Trackball –Pros: Small physical footprint Good for kiosks Joystick –Easy to use, lots of buttons –Good for tracking (guide or follow an on screen object) –Does it map well to your app? Touchpoint –Pressure-sensitive ‘nubbin’ on laptops –Keep fingers on the home position

25. 1-25 © 2010 Pearson Addison-Wesley. All rights reserved. Indirect-Control Pointing Touchpad –Laptop mouse device –Lack of moving parts, and low profile –Accuracy, esp. those w/ motor disabilities Graphics Tablet –Screen shot –comfort –good for cad, artists –Limited data entry

26. 1-26 © 2010 Pearson Addison-Wesley. All rights reserved. Indirect pointing devices mouse –the hand rests in a comfortable position, buttons on the mouse are easily pressed, even long motions can be rapid, and positioning can be precise trackball –usually implemented as a rotating ball 1 to 6 inches in diameter that moves a cursor joystick –are appealing for tracking purposes graphics tablet –a touch-sensitive surface separate from the screen touchpad –built-in near the keyboard offers the convenience and precision of a touchscreen while keeping the user's hand off the display surface 8-26

27. 1-27 © 2010 Pearson Addison-Wesley. All rights reserved. Comparison of pointing devices Human-factors variables –speed of motion for short and long distances –accuracy of positioning –error rates –learning time –user satisfaction Other variables –cost –durability –space requirements –weight –left- versus right-hand use –likelihood to cause repetitive-strain injury –compatibility with other systems 8-27

28. 1-28 © 2010 Pearson Addison-Wesley. All rights reserved. Comparison of pointing devices (cont.) Some results –direct pointing devices faster, but less accurate –graphics tablets are appealing when user can remain with device for long periods without switching to keyboard –mouse is faster than isometric joystick –for tasks that mix typing and pointing, cursor keys a faster and are preferred by users to a mouse –muscular strain is low for cursor keys Fitts' Law –Index of difficulty = log2 (2D / W) –Time to point = C1 + C2 (index of difficulty) –C1 and C2 and constants that depend on the device –Index of difficulty is log2 (2*8/1) = log2(16) = 4 bits –A three-component equation was thus more suited for the high-precision pointing task: –Time for precision pointing = C1 + C2 (index of difficulty) + C3 log2 (C4 / W) 8-28

29. 1-29 © 2010 Pearson Addison-Wesley. All rights reserved. Fitts’s Law Paul Fitts (1954) developed a model of human hand movement Used to predict time to point at an object What are the factors to determine the time to point to an object? –D – distance to target –W – size of target Just from your own experience, is this function linear? –No, since if Target A is D distance and Target B is 2D distance, it doesn’t take twice as long –What about target size? Not linear there either MT = a + b log 2 (D/W + 1) –a = time to start/stop in seconds (empirically measured per device) –b = inherent speed of the device (empirically measured per device) –Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm Ans: 300 + 200 log 2 (14/2 + 1) = 900 ms –Really a slope-intercept model

30. 1-30 © 2010 Pearson Addison-Wesley. All rights reserved. Very Successfully Studied Applies to –Feet, eye gaze, head mounted sights –Many types of input devices –Physical environments (underwater!) –User populations (even retarded and drugged) –Drag & Drop and Point & Click Limitations –Dimensionality –Software accelerated pointer motion –Training –Trajectory Tasks (Accot-Zhai Steering Law) –Decision Making (Hick’s Law) Results (what does it say about) –Buttons and widget size? –Edges? –Popup vs. pull-down menus –Pie vs. Linear menus –iPhone/web pages (real borders) vs. monitor+mouse (virtual borders) Interesting readings: –http://particletree.com/features/visualizing-fittss-law/http://particletree.com/features/visualizing-fittss-law/ –http://www.asktog.com/columns/022DesignedToGiveFitts.htmlhttp://www.asktog.com/columns/022DesignedToGiveFitts.html –http://www.yorku.ca/mack/GI92.html

31. 1-31 © 2010 Pearson Addison-Wesley. All rights reserved. Novel devices 1.Foot controls 2.Eye-tracking 3.Multiple-degrees-of-freedom devices 4.DataGlove 5.Haptic feedback 6.Bimanual input 7.Ubiquitous computing and tangible user interfaces 8.Handheld devices 9.Smart pens 10.Table top touch screens 11.Game controllers 8-31

32. 1-32 © 2010 Pearson Addison-Wesley. All rights reserved. Novel devices (cont.) 8-32

33. 1-33 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Themes: –Make device more diverse Users Task –Improve match between task and device –Improve affordance –Refine input –Feedback strategies Foot controls –Already used in music where hands might be busy –Cars –Foot mouse was twice as slow as hand mouse –Could specify ‘modes’

34. 1-34 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Eye-tracking –Accuracy 1-2 degrees –selections are by constant stare for 200-600 ms –How do you distinguish w/ a selection and a gaze? –Combine w/ manual input Multiple degree of freedom devices –Logitech Spaceball and SpaceMouse –Ascension Bird –Polhemus Liberty and IsoTrack

35. 1-35 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Boom Chameleon –Pros: Natural, good spatial understanding –Cons: limited applications, hard to interact (very passive) DataGlove –Pinch glove –Gesture recognition –American Sign Language, musical director –Pros: Natural –Cons: Size, hygiene, accuracy, durability

36. 1-36 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Haptic Feedback –Why is resistance useful? –SensAble Technology’s Phantom –Cons: limited applications –Sound and vibration are easier and can be a good approximation Rumble pack Two-Handed input –Different hands have different precision –Non-dominant hand selects fill, the other selects objects Ubiquitous Computing and Tangible User Interface –Active Badges allows you to move about the house w/ your profile –Which sensors could you use? –Elderly, disabled –Research: Smart House –Myron Kruger – novel user participation in art (Lots of exhibit art at siggraph)

37. 1-37 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Paper/Whiteboards –Video capture of annotations –Record notes (special tracked pens Logitech digital pen) Handheld Devices –PDA –Universal remote –Help disabled Read LCD screens Rooms in building Maps –Interesting body-context-sensitive. Ex. hold PDA by ear = phone call answer.

38. 1-38 © 2010 Pearson Addison-Wesley. All rights reserved. Novel Devices Miscellaneous –Shapetape – reports 3D shape. Tracks limbs Engineer for specific app (like a gun trigger connected to serial port) –Pros: good affordance –Cons: Limited general use, time

39. 1-39 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces Speech recognition still does not match the fantasy of science fiction: –demands of user's working memory –background noise problematic –variations in user speech performance impacts effectiveness –most useful in specific applications, such as to benefit handicapped users NAVI -- Navigational Aids for the Visually Impaired 2:29Navigational Aids for the Visually Impaired 8-39

40. 1-40 © 2010 Pearson Addison-Wesley. All rights reserved. Interaction Performance 60s vs. Today –Performance Hz -> GHz –Memory k -> GB –Storage k -> TB –Input punch cards -> Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web cams –Output 10 character/sec Megapixel displays, color laser, surround sound, force feedback, VR Substantial bandwidth increase!

41. 1-41 © 2010 Pearson Addison-Wesley. All rights reserved. Interaction Performance Future? –Gestural input –Two-handed input –3D I/O –Others: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedback –Engineering research! –Entire companies created around one single technology Current trend: –Multimodal (using car navigation via buttons or voice) –Helps disabled (esp. those w/ different levels of disability)

42. 1-42 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) 8-42

43. 1-43 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) Discrete word recognition –recognize individual words spoken by a specific person; can work with 90- to 98-percent reliability for 20 to 200 word vocabularies –Speaker-dependent training, in which the user repeats the full vocabulary once or twice –Speaker-independent systems are beginning to be reliable enough for certain commercial applications –been successful in enabling bedridden, paralyzed, or otherwise disabled people –also useful in applications with at least one of the following conditions: speaker's hands are occupied mobility is required speaker's eyes are occupied harsh or cramped conditions preclude use of keyboard –voice-controlled editor versus keyboard editor lower task-completion rate lower error rate –use can disrupt problem solving 8-43

44. 1-44 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) Continuous-speech recognition –Not generally available: difficulty in recognizing boundaries between spoken words normal speech patterns blur boundaries many potentially useful applications if perfected Speech store and forward –Voice mail users can receive messages replay messages reply to caller forward messages to other users, delete messages archive messages Systems are low cost and reliable. 8-44

45. 1-45 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) Voice information systems –Stored speech commonly used to provide information about tourist sites, government services, after-hours messages for organizations –Low cost –Voice prompts –Deep and complex menus frustrating –Slow pace of voice output, ephemeral nature of speech, scanning and searching problems –Voice mail –Handheld voice recorders –Audio books –Instructional systems 8-45

46. 1-46 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) Speech generation –Michaelis and Wiggins (1982) suggest that speech generation is "frequently preferable" under these circumstances: The message is simple. The message is short. The message will not be referred to later. The message deals with events in time. The message requires an immediate response. The visual channels of communication are overloaded. The environment is too brightly lit, too poorly lit, subject to severe vibration, or otherwise unsuitable for transmission of visual information. The user must be free to move around. The user is subjected to high G forces or anoxia 8-46

47. 1-47 © 2010 Pearson Addison-Wesley. All rights reserved. Speech and auditory interfaces (cont.) Audio tones, audiolization, and music –Sound feedback can be important: to confirm actions offer warning for visually-impaired users music used to provide mood context, e.g. in games can provide unique opportunities for user, e.g. with simulating various musical instruments 8-47

48. 1-48 © 2010 Pearson Addison-Wesley. All rights reserved. Non-speech Auditory Interface Terms: –Auditory icons – familiar sounds (record real world sound and play it in your app) –Earcons – new learned sounds (door ajar) Role in video games is huge –Emotions, Tension, set mood To create 3D sound –Need to do more than stereo –Take into account Head- related transfer function (HRTF) Ear and head shape New musical instruments –Theremin New ways to arrange music

49. 1-49 © 2010 Pearson Addison-Wesley. All rights reserved. Displays Primary Source of feedback Properties: –Physical Dimension –Resolution –Color Depth and correctness –Brightness, contrast, glare –Power –Refresh rate –Cost –Reliability –# of users

50. 1-50 © 2010 Pearson Addison-Wesley. All rights reserved. Display Technology Monochrome displays (single color) –Low cost –Greater intensity range (medical) Color –Raster Scan CRT –LCD – thin, bright –Plasma – very bright, thin –LED – large public displays –Electronic Ink – new product w/ tiny capsules of negative black particles and positive white –Braille – refreshable cells with dots that rise up

51. 1-51 © 2010 Pearson Addison-Wesley. All rights reserved. Large Displays Multiple Desktop Displays –Multiple CRTs or Flat panels for large desktops –Cheap –Familiar –Spatial divide up tasks –Comparison tasks are easier –Too much info? HMD Eventually -> Every surface a pixel

52. 1-52 © 2010 Pearson Addison-Wesley. All rights reserved. Mobile device displays Applications –Personal Reprogrammable picture frames –Digital family portrait (GaTech) –Business PDAs, cellphones –Medical Monitor patients –Research: Modality Translation Services (Trace Center – University of Wisconsin) As you move about it auto converts data, info, etc. for you

53. 1-53 © 2010 Pearson Addison-Wesley. All rights reserved. Mobile device displays Actions on mobile devices –Monitor information and alert (calendar) –Gather then spread out information (phone) –Participate in groups and relate to individual (networked devices) –Locate services and identify objects (GPS car system) –Capture and then share info (phone)

54. 1-54 © 2010 Pearson Addison-Wesley. All rights reserved. Animation, Image, and Video Content quality has also greatly increased 3D rendering is near life-like Digital Photography is common Scanned documents Video compression Multimedia considerations for the disabled Printers –3D Printers create custom objects from 3D models

55. 1-55 © 2010 Pearson Addison-Wesley. All rights reserved. Displays – Small and Large The display has become the primary source of feedback to the user from the computer –The display has many important features, including: Physical dimensions (usually the diagonal dimension and depth) Resolution (the number of pixels available) Number of available colors, color correctness Luminance, contrast, and glare Power consumption Refresh rates (sufficient to allow animation and video) Cost Reliability 8-55

56. 1-56 © 2010 Pearson Addison-Wesley. All rights reserved. Displays – Small and Large (cont.) Usage characteristics distinguish displays: Portability Privacy Saliency Ubiquity Simultaneity 8-56

57. 1-57 © 2010 Pearson Addison-Wesley. All rights reserved. Displays – Large and Small (cont.) Large displays –Informational wall displays –Interactive wall displays –Multiple desktop displays 8-57