1. How do we communicate with computers?
Dialog design
How do we communicate with computers?

2. Agenda Syllabus modification Part 1 due next week!
Interaction modalities and styles

3. Dialog Design
How does a user interact with the interface?

4. Interaction Styles 1. Command languages
2. WIMP - Window, Icon, Menu, Pointer
3. Direct manipulation
4. Speech/Natural language
5. Pen & Mobile

5. General Issues in Choosing Dialogue Style
Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required

6. Command Languages Earliest UI interaction paradigm
Examples: MS-DOS shell, UNIX, Linux
Little or nothing is visible so…
Work primarily by recall, not recognition
Heavy memory load
Poor choice for novices but...

7. CL Advantages Advantages for experts Speed, conciseness
% ls (hard to beat)
Can express actions beyond a limited set
Flags, piping one command to another
Repetition, extensibility
Scripting, macros
Easier implementation, less overhead
Power
Abstraction, wild cards
Command languages are often maligned (for good reason)
But increased functionality can win out over bad UI (e.g., UNIX)
Try to get both
Avoid excess functionality (comes at cost)

8. CL Dangers With added power, comes added responsibility and danger
UNIX
% rm -r *
Deletes every file that you have, and you can’t get them back

9. CL Design Goals Consistency Good naming and abbreviations
Have options and arguments expressed the same way everywhere
Good naming and abbreviations
UNIX fails here because commands were developed by lots of different people at different organizations
No guidelines provided
Don’t use abbreviations in output
Use standard rule for abbreviations.
Usually fixed length is best

10. General Issues - CL Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required

11. Direct Manipulation Definition:
1) Continuous visibility of the objects and actions of interest
2) Rapid, reversible, incremental actions whose effect is immediately noticeable
3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.)
Shneiderman ‘82

12. DM Essence Representation of reality that can be manipulated
The user is able to apply intellect directly to the task
The tool itself seems to disappear
Related to
Directness
Engagement
Promoted by
Unobtrusive interface
Minimizing gulfs of execution and evaluation
Appropriately responsive system
In end, must characterize direct manipulation by feeling of directness and illusion of manipulating objects at hand

13. Direct Manipulation See pgs 171-174 in DFAB Examples
WYSIWYG editors and word processors
VISICALC - 1st electronic spreadsheet
CAD
Desktop metaphor
Video games

14. Example: Homefinder

15. DM Issues
Advantages
Flexible, easily reversible actions helps reduce anxiety in users
Easy to learn and remember
WYSIWYG provides context & instant visual feedback
Exploits human use of visual spatial cues
Limits types of errors that can be made
Disadvantages
visual representation not self-explanatory, no prompts
Mouse ops may be slower than typing
Screen space intensive
Not good at:
Repetition
History keeping
Certain tasks (Change all italics to bold)
Abstract elements (variables)

16. General Issues - DM Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required
Easier to learn and remember, esp. novices

17. WIMP Windows, Icons, Menus, Pointers
Focus: Menus, Buttons, Forms
Predominant interface paradigm now (with some direct manipulation added)
Advantages: ?

18. Menus Advantages: Disadvantages:
1 keystroke or mouse operation vs. many
No memorization of commands
Limited input set
Disadvantages:
Less direct user control - have to find correct menu / menu item
Not so readily extensible
Slower than keyboarding for experienced users, at least without accelerators
Many different types
pop-up
pull-down
radio buttons
pie buttons
hierarchies

19. Menu Items Various types: Organization strategies Flat Cascading
Pop-up (contextual)
Organization strategies
Create groups of logically similar items
Cover all possibilities
Ensure that items are non-overlapping
Keep wording concise, understandable

20. A Good Example Logical grouping Visual separation of groups
Disabled items “grayed out”
Shortcuts shown
… indicates leads to dialogue

21. Presentation Sequence
Forms, dialogue boxes, menus
Use natural if available
Time
e.g. Breakfast, Lunch, Dinner
Numeric ordering
e.g. Point sizes for font
Other possibilities:
Alphabetical
Group related items
Frequently used first
Most important first
Novices actually do best with alphabetical, then functional, then random
Experts prefer categorization

22. Pointers Allows users to specify location parameter of a command
Provides for some of that direct manipulation
Put this file HERE
Also displays system state info:
Tracking
Busy
Hints
Modes (text cursor vs. selection pointer)

23. Other WIMP issues Windows management Icons Toolbars
How to locate, move, find
Transfer information between
Switch attention between
Icons
Need graphic design attention
Toolbars

24. Newer icons

25. Simple icons

26. Simple icons

27. General Issues - WIMP Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required

28. Natural input Universal design
Take advantage of familiarity, existing knowledge
Alternative input & output
Multi-modal interfaces
Getting “off the desktop”

29. Speech dialogue Why use it? Hands busy Mobility required Eyes occupied
Conditions preclude use of keyboard
Visual impairment
Physical limitation

30. Speech Input Speaker recognition Speech recognition
Tell which person it is (voice print)
Monitoring, recording
Speech recognition
Identify words
IBM ViaVoice, Dragon Dictate, ...
Natural language understanding
(does not necessarily involve audio)
Speech not equal to natural language understanding

31. Recognition Dimensions
Speaker dependent/independent
Parametric patterns are sensitive to speaker
With training (dependent) can get better
Vocabulary
Some have 50,000+ words
Isolated word vs. continuous speech
Continuous: where words stop & begin
Typically a pattern match, no context used
Did you vs.
Didja

32. Recognition Example: My flight experience uh,
I started off as a private
pilot and then I spent 20
years flying as a navigator
on C130s in the air force
and I currently fly as a
first officer on the SAAB
340 turboprop.
my flight experience a and
I started off as a private
pilot and I spent 20
years flying is a navigator
on C-130s in the Air Force
and I currently fly as a
first officer on the sound
340 turboprop.

33. And another example Here here's one of the thoughts I'd throw in on
that. I'd agree with
Howard that the
combination of checklists
and flows is the way way
to manage things best.
hair and here's one of the
outside troodon and
an eye to agree with
power that the
domination checklists
and flows is the way
that managed things best

34. Errors Systems make four types of errors:
Substitution - one for another
Rejection - detected, but not recognized
Insertion - added
Deletion - not detected
So how do you recover from an error?

35. Natural Language Understanding
Putting meaning to the words
Input might be spoken or typed
Holy grail of Artificial Intelligence problems

36. Eliza - a “Faked” Understanding
Person : Men are all alike. ELIZA : In what way? Person : They're always bugging us about something or other. ELIZA : Can you think of a specific example? Person : Well, my boyfriend made me come here. ELIZA : Your boyfriend made you come here? Person : He says I'm depressed much of the time. ELIZA : I am sorry to hear that you are depressed. Person : It's true. I am unhappy. ELIZA : Do you think coming here will help you not to be unhappy? Person : I need some help, that much seems certain. ELIZA : What would it mean to you if you got some help? Person : Perhaps I could learn to get along with my mother. ELIZA : Tell me more about your family
Weizenbaum, J., "ELIZA -- A computer program for the study of natural language communication between man and machine", Communications of the ACM 9(1):36-45, 1966

37. NL Factors/Terms Syntactic Prosodic Pragmatic Semantic
Grammar or structure
Prosodic
Inflection, stress, pitch, timing
Pragmatic
Situated context of utterance, location, time
Semantic
Meaning of words

38. SR/NLU Issues Advantages Easy to learn and remember Powerful
Fast, efficient (not always)
Little screen real estate
Disadvantages
Assumes domain knowledge
Doesn’t work well enough yet
Requires confirmation
And recognition will always be error-prone
Expensive to implement
Unrealistic expectations can generate mistrust

39. Speech Output Tradeoffs in speed, naturalness and understandability
Male or female voice?
Technical issues (freq. response of phone)
User preference (depends on the application)
Rate of speech
Technically up to 550 wpm!
Depends on listener
Synthesized or Pre-recorded?
Synthesized: Better coverage, flexibility
Recorded: Better quality, acceptance

40. Speech Output Synthesis Recorded segments
Quality depends on software ($$)
Influence of vocabulary and phrase choices
Recorded segments
Store tones, then put them together
The transitions are difficult (e.g., numbers)

41. Designing Speech Interaction
Constrain vocabulary
Limit valid commands
Structure questions wisely (Yes/No)
Manage the interaction
Examples?
Slow speech rate, but concise phrases
Design for failsafe error recovery
Visual record of input/output
Design for the user – Wizard of Oz

42. Speech Tools/Toolkits
Java Speech SDK
FreeTTS
IBM JavaBeans for speech
Microsoft speech SDK (Visual Basic, etc.)
OS capabilities (speech recognition and synthesis built in to OS) (TextEdit)
VoiceXML

43. General Issues – Speech/NL
Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required

44. Non-speech audio
Good for indicating changes, since we ignore continuous sounds
Traditionally used for warnings, alarms or status information
Provides secondary representation
Supports visual interface
Provides information that helps reduce error
Tradeoff in using natural (real) sounds vs. synthesized noises.

45. Non-speech audio examples
Error ding
Info beep
arriving ding
Recycle
Battery critical
Logoff
Logon
Others?

46. Pen, Touch, & Mobile interaction

47. Pen, Touch, & Mobile dialog
Stylus or finger
Tradeoffs of each?
Pen as a standard mouse (doubleclick?)
Variety of platforms
Desktop touch screens or input pads (Wacom)
Tablet PCs
Handheld and Mobile devices
Electronic whiteboards
Platforms often involve variety of size and other constraints

48. Mobile devices More common as more platforms available
PDA
Cell phone
Ultra mobile tablets
GPS
Smaller display (160x160), (320x240)
Few buttons, different interactions
Free-form ink
Soft keyboard
Numeric keyboard => text
Stroke recognition
Hand printing / writing recognition

50. Soft Keyboard Presents a small diagram of keyboard
You click on buttons/keys with pen
QWERTY vs. alphabetical
Tradeoffs?
Alternatives?

51. Numeric Keypad http://www.t9.com/
You press out letters of your word, it matches the most likely word, then gives optional choices
Faster than multiple presses per key
Used in mobile phones

52. Other pen text input Graffiti – Palm mobile devices Experimental
Unistroke recognition
Experimental
Cirrin
World level unistroke
Quickwriting
Harder to learn than graffiti

53. Hand Printing / Writing Recognition
Recognizing letters and numbers and special symbols
Lots of systems (commercial too)
English, kanji, etc.
Not perfect, but people aren’t either!
People - 96% handprinted single characters
Computer - >97% is really good

54. Recognition Issues Boxed vs. Free-Form input Printed vs. Cursive
Sometimes encounter boxes on forms
Printed vs. Cursive
Cursive is much more difficult
Letters vs. Words
Cursive is easier to do in words vs individual letters, as words create more context
Usually requires existence of a dictionary
Real-time vs. off-line

55. Pen Gesture Commands Might mean delete Insert Paragraph
Define a series of (hopefully) simple drawing gestures that mean different commands in a system

56. Pen Use Modes
Often, want a mix of free-form drawing and special commands
How does user switch modes?
Mode icon on screen
Button on pen
Button on device

57. Error Correction Having to correct errors can slow input tremendously
Strategies
Erase and try again (repetition)
When uncertain, system shows list of best guesses (n-best list)
Others??

58. Free-form Ink Ink is the data, take as is
Human is responsible for understanding and interpretation
Often time-stamped
Applications
Signature verification
Notetaking
Electronic whiteboards
Sketching

59. Electronic whiteboards
Smartboard and Mimio
Can integrate with projection
Large surface to interact with
Issues?

60. Touch tables
Which techniques might be similar to smaller touchscreens?
Which would differ?
How similar and different from interactive white boards?
Microsoft Surface

61. Real paper
Anoto digital paper and pen technology (
Other pens available:
Issues?

62. General Issues – Pen input
Who is in control - user or computer
Initial training required
Learning time to become proficient
Speed of use
Generality/flexibility/power
Special skills - typing
Screen space required
Computational resources required

63. Other interesting interactions
Gesture input
Wii
Lots of other specialized hardware for tracking
3D interaction
Stereoscopic displays
Virtual reality
Immersive displays such as glasses, caves
Augmented reality
Head trackers and vision based tracking
Tangible interaction
Use physical objects to express input