Dialog Design Command languages, direct manipulation, and WIMP

Part 2 continued  Keep design materials around Discussion of design space exploration in writeup  Choose one design and iterate on details Explain entire design in some way Create interactive prototype  Evaluation plan

Dialog Design  How does a user interact with the interface?

Dialog Styles  1. Command languages  2. WIMP - Window, Icon, Menu, Pointer  3. Direct manipulation  4. Speech/Natural language  5. Gesture, pen, VR

General Issues in Dialogue Style  Who is in control - user or computer  Initial training required  Learning time to become proficient  Speed of use  Generality/flexibility/power  Special skills - typing  Gulf of evaluation / gulf of execution  Screen space required  Computational resources required

Command Languages  Earliest UI interaction paradigm  Examples: MS-DOS shell, UNIX, Linux  Little or nothing is visible so… Work primarily by recall, not recognition Heavy memory load  Poor choice for novices but...

CL Advantages  Advantages for experts Speed, conciseness % ls (hard to beat) Can express actions beyond a limited set  Flags, piping one command to another Repetition, extensibility  Scripting, macros Easier implementation, less overhead Power  Abstraction, wild cards

CL Dangers  With added power, comes added responsibility and danger UNIX % rm -r *  Deletes every file that you have, and you can’t get them back

CL Design Goals  Consistency Have options and arguments expressed the same way everywhere  Good naming and abbreviations  UNIX fails here because commands were developed by lots of different people at different organizations  No guidelines provided

Names and Abbreviations  Specificity versus Generality General words  More familiar, easier to accept Specific (typically better)  More descriptive, meaningful, distinctive (Nonsense does surprisingly well in small set)  Abbrevs. allow for faster actions Expert performance begins to be dominated by motor times such as # of keystrokes Not good idea for novices (Allow but don’t require)

General Issues - CL  Who is in control - user or computer  Initial training required  Learning time to become proficient  Speed of use  Generality/flexibility/power  Special skills - typing  Gulf of evaluation / gulf of execution  Screen space required  Computational resources required

Dialog Design  1. Command language  2. Direct manipulation  3. WIMP  4. Speech/Natural language  5. Gesture, pen, VR

Direct Manipulation Essence  Representation of reality that can be manipulated  The user is able to apply intellect directly to the task  The tool itself seems to disappear

Direct Manipulation Definition:  1) Continuous visibility of the objects and actions of interest  2) Rapid, reversible, incremental actions whose effect is immediately noticeable  3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.) Shneiderman ‘82

Direct Manipulation  Examples WYSIWYG editors and word processors VISICALC - 1 st electronic spreadsheet CAD Desktop metaphor Video games

Example: Homefinder

Direct Manipulation  Advantages  Disadvantages

DM Advantages  Easier to learn & remember, particularly for novices  Flexible, easily reversible actions helps reduce anxiety in users  Provides context & instant visual feedback so user can tell if objectives are being achieved  Exploits human use of visual spatial cues  Limits types of errors that can be made

DM Problems  Screen space intensive (info not all that dense)  Need to learn meaning of components of visual representation  Visual representation may be misleading  Mouse ops may be slower than typing  Not self-explanatory (no prompts)

DM Problems  Not good at Repetition History keeping (harder) Certain tasks (Change all italics to bold) Abstract elements (variables) Macros harder

More Psychological View  What is directness? (not always done well)  Related to two things: Distance Engagement  Unobtrusive and responsive Hutchins, Hollan, Norman ‘86 Goals System Execution Evaluation

General Issues - DM  Who is in control - user or computer  Initial training required  Learning time to become proficient  Speed of use  Generality/flexibility/power  Special skills - typing  Gulf of evaluation / gulf of execution  Screen space required  Computational resources required

Dialog Design  1. Command language  2. Direct manipulation  3. WIMP  4. Speech/Natural language  5. Gesture, pen, VR

WIMP  Windows, Icons, Menus, Pointers Focus: Menus, Buttons, Forms  Predominant interface paradigm now (with some direct manipulation added)  Advantages: ?

Menus  Advantages: 1 keystroke or mouse operation vs. many No memorization of commands Limited input set  Disadvantages: Less direct user control - have to find correct menu / menu item Not so readily extensible Slower than keyboarding for experienced users, at least without accelerators

Menu Items  Organization strategies Create groups of logically similar items Cover all possibilities Ensure that items are non-overlapping Keep wording concise, understandable

A Good Example  Logical grouping  Visual separation of groups  Disabled items “grayed out”  Shortcuts shown  … indicates leads to dialogue

Presentation Sequence  Forms, dialogue boxes, menus  Use natural if available Time  e.g. Breakfast, Lunch, Dinner Numeric ordering  e.g. Point sizes for font  Other possibilities: Alphabetical Group related items Frequently used first Most important first

Other WIMP issues  Windows management How to locate, move, find Transfer information between  Icons Need graphic design attention  Toolbars

General Issues - WIMP  Who is in control - user or computer  Initial training required  Learning time to become proficient  Speed of use  Generality/flexibility/power  Special skills - typing  Gulf of evaluation / gulf of execution  Screen space required  Computational resources required

Natural input  Universal design  Take advantage of familiarity, existing knowledge  Alternative input & output  Multi-modal interfaces  Getting “off the desktop”

When to Use Speech  Hands busy  Mobility required  Eyes occupied  Conditions preclude use of keyboard  Visual impairment  Physical limitation

Speech Input  Speaker recognition Tell which person it is (voice print) Monitoring, recording  Speech recognition Identify words IBM ViaVoice, Dragon Dictate,...  Natural language understanding

Recognition Dimensions  Speaker dependent/independent Parametric patterns are sensitive to speaker With training (dependent) can get better  Vocabulary Some have 50,000+ words  Isolated word vs. continuous speech Continuous: where words stop & begin Typically a pattern match, no context used Did you vs. Didja

Recognition Example: My flight experience uh, I started off as a private pilot and then I spent 20 years flying as a navigator on C130s in the air force and I currently fly as a first officer on the SAAB 340 turboprop. my flight experience a and I started off as a private pilot and I spent 20 years flying is a navigator on C-130s in the Air Force and I currently fly as a first officer on the sound 340 turboprop.

And another example Here here's one of the thoughts I'd throw in on that. I'd agree with Howard that the combination of checklists and flows is the way way to manage things best. hair and here's one of the outside troodon and an eye to agree with power that the domination checklists and flows is the way that managed things best

Errors  Systems make four types of errors: Substitution - one for another Rejection - detected, but not recognized Insertion - added Deletion - not detected  Problems with recovery

Natural Language Understanding  Putting meaning to the words  Input might be spoken or typed  Holy grail of Artificial Intelligence problems

Eliza - a “Faked” Understanding  Person : Men are all alike. ELIZA : In what way? Person : They're always bugging us about something or other. ELIZA : Can you think of a specific example? Person : Well, my boyfriend made me come here. ELIZA : Your boyfriend made you come here? Person : He says I'm depressed much of the time. ELIZA : I am sorry to hear that you are depressed. Person : It's true. I am unhappy. ELIZA : Do you think coming here will help you not to be unhappy? Person : I need some help, that much seems certain. ELIZA : What would it mean to you if you got some help? Person : Perhaps I could learn to get along with my mother. ELIZA : Tell me more about your family   Weizenbaum, J., "ELIZA -- A computer program for the study of natural language communication between man and machine", Communications of the ACM 9(1):36-45, 1966

NL Factors/Terms  Syntactic Grammar or structure  Prosodic Inflection, stress, pitch, timing  Pragmatic Situated context of utterance, location, time  Semantic Meaning of words

SR/NLU Issues Advantages  Easy to learn and remember  Powerful  Fast, efficient (not always)  Little screen real estate Disadvantages  Assumes domain knowledge  Doesn’t work well enough yet Requires confirmation And recognition will always be error-prone  Expensive to implement  Unrealistic expectations can generate mistrust

Speech Output  Tradeoffs in speed, naturalness and understandability  Male or female voice? Technical issues (freq. response of phone) User preference (depends on the application)  Rate of speech Technically up to 550 wpm! Depends on listener  Synthesized or Pre-recorded? Synthesized: Better coverage, flexibility Recorded: Better quality, acceptance

Speech Output  Synthesis Quality depends on software ($$) Influence of vocabulary and phrase choices  Recorded segments Store tones, then put them together The transitions are difficult (e.g., numbers)

Designing Speech Interaction  Constrain vocabulary Limit valid commands Structure questions wisely (Yes/No) Manage the interaction Examples?  Slow speech rate, but concise phrases  Design for failsafe error recovery  Visual record of input/output  Design for the user – Wizard of Oz

Speech Tools/Toolkits  Java Speech SDK FreeTTS  IBM JavaBeans for speech  Microsoft speech SDK (Visual Basic, etc.)  OS capabilities (speech recognition and synthesis built in to OS) (TextEdit)  VoiceXML

General Issues – Speech/NL  Who is in control - user or computer  Initial training required  Learning time to become proficient  Speed of use  Generality/flexibility/power  Special skills - typing  Gulf of evaluation / gulf of execution  Screen space required  Computational resources required