How do we communicate with computers? Dialog design How do we communicate with computers?

Agenda Syllabus modification Part 1 due next week! Interaction modalities and styles

Dialog Design How does a user interact with the interface?

Interaction Styles 1. Command languages 2. WIMP - Window, Icon, Menu, Pointer 3. Direct manipulation 4. Speech/Natural language 5. Pen & Mobile

General Issues in Choosing Dialogue Style Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required

Command Languages Earliest UI interaction paradigm Examples: MS-DOS shell, UNIX, Linux Little or nothing is visible so… Work primarily by recall, not recognition Heavy memory load Poor choice for novices but...

CL Advantages Advantages for experts Speed, conciseness % ls (hard to beat) Can express actions beyond a limited set Flags, piping one command to another Repetition, extensibility Scripting, macros Easier implementation, less overhead Power Abstraction, wild cards Command languages are often maligned (for good reason) But increased functionality can win out over bad UI (e.g., UNIX) Try to get both Avoid excess functionality (comes at cost)

CL Dangers With added power, comes added responsibility and danger UNIX % rm -r * Deletes every file that you have, and you can’t get them back

CL Design Goals Consistency Good naming and abbreviations Have options and arguments expressed the same way everywhere Good naming and abbreviations UNIX fails here because commands were developed by lots of different people at different organizations No guidelines provided Don’t use abbreviations in output Use standard rule for abbreviations. Usually fixed length is best

General Issues - CL Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required

Direct Manipulation Definition: 1) Continuous visibility of the objects and actions of interest 2) Rapid, reversible, incremental actions whose effect is immediately noticeable 3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.) Shneiderman ‘82

DM Essence Representation of reality that can be manipulated The user is able to apply intellect directly to the task The tool itself seems to disappear Related to Directness Engagement Promoted by Unobtrusive interface Minimizing gulfs of execution and evaluation Appropriately responsive system In end, must characterize direct manipulation by feeling of directness and illusion of manipulating objects at hand

Direct Manipulation See pgs 171-174 in DFAB Examples WYSIWYG editors and word processors VISICALC - 1st electronic spreadsheet CAD Desktop metaphor Video games

Example: Homefinder

DM Issues Advantages Flexible, easily reversible actions helps reduce anxiety in users Easy to learn and remember WYSIWYG provides context & instant visual feedback Exploits human use of visual spatial cues Limits types of errors that can be made Disadvantages visual representation not self-explanatory, no prompts Mouse ops may be slower than typing Screen space intensive Not good at: Repetition History keeping Certain tasks (Change all italics to bold) Abstract elements (variables)

General Issues - DM Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required Easier to learn and remember, esp. novices

WIMP Windows, Icons, Menus, Pointers Focus: Menus, Buttons, Forms Predominant interface paradigm now (with some direct manipulation added) Advantages: ?

Menus Advantages: Disadvantages: 1 keystroke or mouse operation vs. many No memorization of commands Limited input set Disadvantages: Less direct user control - have to find correct menu / menu item Not so readily extensible Slower than keyboarding for experienced users, at least without accelerators Many different types pop-up pull-down radio buttons pie buttons hierarchies

Menu Items Various types: Organization strategies Flat Cascading Pop-up (contextual) Organization strategies Create groups of logically similar items Cover all possibilities Ensure that items are non-overlapping Keep wording concise, understandable

A Good Example Logical grouping Visual separation of groups Disabled items “grayed out” Shortcuts shown … indicates leads to dialogue

Presentation Sequence Forms, dialogue boxes, menus Use natural if available Time e.g. Breakfast, Lunch, Dinner Numeric ordering e.g. Point sizes for font Other possibilities: Alphabetical Group related items Frequently used first Most important first Novices actually do best with alphabetical, then functional, then random Experts prefer categorization

Pointers Allows users to specify location parameter of a command Provides for some of that direct manipulation Put this file HERE Also displays system state info: Tracking Busy Hints Modes (text cursor vs. selection pointer)

Other WIMP issues Windows management Icons Toolbars How to locate, move, find Transfer information between Switch attention between Icons Need graphic design attention Toolbars

Newer icons

Simple icons

Simple icons

General Issues - WIMP Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required

Natural input Universal design Take advantage of familiarity, existing knowledge Alternative input & output Multi-modal interfaces Getting “off the desktop”

Speech dialogue Why use it? Hands busy Mobility required Eyes occupied Conditions preclude use of keyboard Visual impairment Physical limitation

Speech Input Speaker recognition Speech recognition Tell which person it is (voice print) Monitoring, recording Speech recognition Identify words IBM ViaVoice, Dragon Dictate, ... Natural language understanding (does not necessarily involve audio) Speech not equal to natural language understanding

Recognition Dimensions Speaker dependent/independent Parametric patterns are sensitive to speaker With training (dependent) can get better Vocabulary Some have 50,000+ words Isolated word vs. continuous speech Continuous: where words stop & begin Typically a pattern match, no context used Did you vs. Didja

Recognition Example: My flight experience uh, I started off as a private pilot and then I spent 20 years flying as a navigator on C130s in the air force and I currently fly as a first officer on the SAAB 340 turboprop. my flight experience a and I started off as a private pilot and I spent 20 years flying is a navigator on C-130s in the Air Force and I currently fly as a first officer on the sound 340 turboprop.

And another example Here here's one of the thoughts I'd throw in on that. I'd agree with Howard that the combination of checklists and flows is the way way to manage things best. hair and here's one of the outside troodon and an eye to agree with power that the domination checklists and flows is the way that managed things best

Errors Systems make four types of errors: Substitution - one for another Rejection - detected, but not recognized Insertion - added Deletion - not detected So how do you recover from an error?

Natural Language Understanding Putting meaning to the words Input might be spoken or typed Holy grail of Artificial Intelligence problems

Eliza - a “Faked” Understanding Person : Men are all alike. ELIZA : In what way? Person : They're always bugging us about something or other. ELIZA : Can you think of a specific example? Person : Well, my boyfriend made me come here. ELIZA : Your boyfriend made you come here? Person : He says I'm depressed much of the time. ELIZA : I am sorry to hear that you are depressed. Person : It's true. I am unhappy. ELIZA : Do you think coming here will help you not to be unhappy? Person : I need some help, that much seems certain. ELIZA : What would it mean to you if you got some help? Person : Perhaps I could learn to get along with my mother. ELIZA : Tell me more about your family http://www-ai.ijs.si/eliza/eliza.html Weizenbaum, J., "ELIZA -- A computer program for the study of natural language communication between man and machine", Communications of the ACM 9(1):36-45, 1966

NL Factors/Terms Syntactic Prosodic Pragmatic Semantic Grammar or structure Prosodic Inflection, stress, pitch, timing Pragmatic Situated context of utterance, location, time Semantic Meaning of words

SR/NLU Issues Advantages Easy to learn and remember Powerful Fast, efficient (not always) Little screen real estate Disadvantages Assumes domain knowledge Doesn’t work well enough yet Requires confirmation And recognition will always be error-prone Expensive to implement Unrealistic expectations can generate mistrust

Speech Output Tradeoffs in speed, naturalness and understandability Male or female voice? Technical issues (freq. response of phone) User preference (depends on the application) Rate of speech Technically up to 550 wpm! Depends on listener Synthesized or Pre-recorded? Synthesized: Better coverage, flexibility Recorded: Better quality, acceptance

Speech Output Synthesis Recorded segments Quality depends on software ($$) Influence of vocabulary and phrase choices http://www.research.att.com/~ttsweb/tts/demo.php#top Recorded segments Store tones, then put them together The transitions are difficult (e.g., numbers)

Designing Speech Interaction Constrain vocabulary Limit valid commands Structure questions wisely (Yes/No) Manage the interaction Examples? Slow speech rate, but concise phrases Design for failsafe error recovery Visual record of input/output Design for the user – Wizard of Oz

Speech Tools/Toolkits Java Speech SDK FreeTTS 1.1.1 http://freetts.sourceforge.net/docs/index.php IBM JavaBeans for speech Microsoft speech SDK (Visual Basic, etc.) OS capabilities (speech recognition and synthesis built in to OS) (TextEdit) VoiceXML

General Issues – Speech/NL Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required

Non-speech audio Good for indicating changes, since we ignore continuous sounds Traditionally used for warnings, alarms or status information Provides secondary representation Supports visual interface Provides information that helps reduce error Tradeoff in using natural (real) sounds vs. synthesized noises.

Non-speech audio examples Error ding Info beep Email arriving ding Recycle Battery critical Logoff Logon Others?

Pen, Touch, & Mobile interaction

Pen, Touch, & Mobile dialog Stylus or finger Tradeoffs of each? Pen as a standard mouse (doubleclick?) Variety of platforms Desktop touch screens or input pads (Wacom) Tablet PCs Handheld and Mobile devices Electronic whiteboards Platforms often involve variety of size and other constraints

Mobile devices More common as more platforms available PDA Cell phone Ultra mobile tablets GPS Smaller display (160x160), (320x240) Few buttons, different interactions Free-form ink Soft keyboard Numeric keyboard => text Stroke recognition Hand printing / writing recognition

http://www.blackberry.com/ http://www.oqo.com/

Soft Keyboard Presents a small diagram of keyboard You click on buttons/keys with pen QWERTY vs. alphabetical Tradeoffs? Alternatives?

Numeric Keypad http://www.t9.com/ You press out letters of your word, it matches the most likely word, then gives optional choices Faster than multiple presses per key Used in mobile phones http://www.t9.com/

Other pen text input Graffiti – Palm mobile devices Experimental Unistroke recognition Experimental Cirrin World level unistroke Quickwriting Harder to learn than graffiti

Hand Printing / Writing Recognition Recognizing letters and numbers and special symbols Lots of systems (commercial too) English, kanji, etc. Not perfect, but people aren’t either! People - 96% handprinted single characters Computer - >97% is really good

Recognition Issues Boxed vs. Free-Form input Printed vs. Cursive Sometimes encounter boxes on forms Printed vs. Cursive Cursive is much more difficult Letters vs. Words Cursive is easier to do in words vs individual letters, as words create more context Usually requires existence of a dictionary Real-time vs. off-line

Pen Gesture Commands Might mean delete Insert Paragraph Define a series of (hopefully) simple drawing gestures that mean different commands in a system

Pen Use Modes Often, want a mix of free-form drawing and special commands How does user switch modes? Mode icon on screen Button on pen Button on device

Error Correction Having to correct errors can slow input tremendously Strategies Erase and try again (repetition) When uncertain, system shows list of best guesses (n-best list) Others??

Free-form Ink Ink is the data, take as is Human is responsible for understanding and interpretation Often time-stamped Applications Signature verification Notetaking Electronic whiteboards Sketching

Electronic whiteboards Smartboard and Mimio Can integrate with projection Large surface to interact with Issues? http://www.mimio.com/ http://www.smarttech.com/

Touch tables Which techniques might be similar to smaller touchscreens? Which would differ? How similar and different from interactive white boards? Microsoft Surface

Real paper Anoto digital paper and pen technology (http://www.anoto.com/) Other pens available: Issues? http://www.logitech.com/ http://www.epos-ps.com/

General Issues – Pen input Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Screen space required Computational resources required

Other interesting interactions Gesture input Wii Lots of other specialized hardware for tracking 3D interaction Stereoscopic displays Virtual reality Immersive displays such as glasses, caves Augmented reality Head trackers and vision based tracking Tangible interaction Use physical objects to express input