Direct Manipulation Shneiderman and Plaisant, Chapter 6 (from slides at Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct Manipulation interfaces. Human-Computer Interaction, 1,

Overview Introduction –Big picture first, details and a theoretical account later Examples of Direct-Manipulation systems Explanations of Direct Manipulation –Hutchins et al. paper Virtual Environments, or Immersive Interfaces

Overview Have talked about: –History Discontinuities in interface styles, –Switches-terminals/command-graphic- ??? History may not repeat, but it rhymes … sure looks similar –Moore’s law Exponential change results in discontinuities, “disruptive” –“Usability” and evaluation techniques E.g., heuristic evaluation, user testing –Design Process (spiral) –Design Sequence: Select interaction styles –Design Refinement through evaluation Recall, “radical transformation” and design –Now, will look at each interaction style in a bit of detail Direct manipulation, command line, menu, form … mainly direct manipulation

Recall, Interaction Styles from Shneiderman Styles for “giving commands” to the system 5 main types (p. 67): –Direct Manipulation More detail later … now! –Menu selection –Form fillin –Command language –Natural language May blend, especially when users are diverse

Trick question: When was the first direct manipulation interface?

First Direct Manipulation Interface? Sutherland’s Sketchpad, 1963 Surely, graphic user interface User can draw figures directly –On screen with lightpen Direct manipulation: –“Visually presents task concepts” Graphic user interface allowing visual representation of task From 1 st demos to ubiquity –20 years (1983) Early widespread use (Mac) –40 years (2003) “Everywhere” - MS windows, … Ubiquitous in 50 – mobile devices

Less trick question: What allowed the ubiquity of direct manipulation interfaces?

Paradigm Example: File System Direct Manipulation vs. Command Line Direct Manipulation: –User operates on model world to change task world Will sharpen this up with more detailed accounts –Dix et al. and Hutchins, Hollan, & Norman (reading)

“Direct Manipulation” Interface to Car No Computer, but “Direct” – “an ultimate example” Task: Maneuver the car from Point A to Point B (within certain constraints) Interface: Steering wheel, pedals, etc. Geographic area through which user (driver) maneuvers car is displayed directly in front of user (and in high resolution, and in large field of view!) Manipulating steering wheel top to left (and right) causes car front to move left (and right) visual and kinetic feedback is immediate Pushing brake pedal results in car slowing arbitrary mapping of user action to car maneuver (automatized) visual and kinetic feedback is immediate Definition, again: User operates on a visually displayed “world” to perform tasks in a domain Here, the world on which the user operates, is the real world For computing tasks, there is the task world and an interface model world The user operates on the model world to change the task world

Recall (or not), from 1 st class Interaction Frameworks (Dix) Interaction: –Communication between user and system Why have a framework? –Allows “precision” in accounting for differences E.g., gulfs of execution and evaluation –Presents global view All elements receive attention Task “work on task” “commands” System User “gives” “performs” “feedback”

Background: Models in UI Design From Dix et al. Models of UI design consider: –System –Interface –User Will “rewrite” Dix’s account to incorporate standard account of models Task “work on task” “commands” System User “gives” “performs” “feedback” and its representation

Background: Models in UI Design From Dix et al. Models of UI design consider: –System –Interface –User Will “rewrite” Dix’s account to incorporate standard account of models First, it is the System that directly provides feedback about the task Task “work on task” “gives commands” User feedback and its representation System Task “work on task” “commands” System User “gives” “performs” “feedback” and its representation

Models in UI Design Next, just change notation a bit And … Task “work on task” “gives commands” User feedback and its representation System Task “work on task” “gives commands” User System

Models in UI Design For our purposes, Dix’s System (which also represents task) is comprised of two components 1. System model –How system actually works Implementation model 2. Interface model –Model system presents to user Manifest model Task “work on task” “gives commands” User System Model Interface Model Finally, 3. User model –How the user thinks system works –Conceptual model User Model

User/Conceptual/Mental … Model How the user thinks system works –Conceptual model May or may not correspond to the system! Ideally, should match closely, … if (and only if) system reflects task Mental model of the system (and task)

Recall, Mental Model People have “mental models” of how things work (including computer systems), which are built from: –affordances –causality –constraints –mapping –positive transfer –population stereotypes/cultural standards –instructions –Interactions Models allow people to mentally simulate operation of device Models may be wrong –particularly if above attributes are misleading Looked at each of the above

Example of DM: Word Processing People used to use script files for formatting, until 1980’s –E.g., nroff “word processors” allowed direct manipulation of document (text and images) output –Albeit with a fair amount of pointing, selecting, and menu use

Command-line (!) Editors “Line oriented” Above command, *1,5L lists those lines from the file being edited

Example of DM: Word Processing What You See Is What You Get (WYSIWYG) Mid 80’s: WYSIWYG –Apple Mac, Word, Corel’s WordPerfect –See full page of text, as it will appear –Cursor action is visible Attention focus – less effort –Cursor motion is natural Arrow/mouse vs. ‘Up 6’ no converting – less effort –Labeled icons make actions rapid Remind users of possible actions – less effort –Immediate display action’s results Know what happened, feedback – less effort –Rapid Response and Display “Sense of power” and – less effort –Reversible Actions Lowers anxiety, easy recovery – less effort

Example of DM: Word Processing What You See Is What You Get (WYSIWYG) Mid 80’s: WYSIWYG –Apple Mac, Word, Corel’s WordPerfect –See full page of text, as it will appear –Cursor action is visible Attention focus – less effort –Cursor motion is natural Arrow/mouse vs. ‘Up 6’ no converting – less effort –Labeled icons make actions rapid Remind users of possible actions – less effort –Immediate display action’s results Know what happened, feedback – less effort –Rapid Response and Display “Sense of power” and – less effort –Reversible Actions Lowers anxiety, easy recovery – less effort

Less Effort is Good!!! Will see this as the core of Hutchins et al.’s characterization of direct manipulation

Example of DM: VisiCalc and Later People used to use paper and pencil for accounting, until 1980’s! Shneiderman –VisiCalc spreadsheet (1979) and its descendants –Imagine erasing a thousand numbers … “VisiCalc users delighted in watching the program propagate changes across the screen.” –In some cases, spatial representations provide a better model of reality –Successful spatial data- management systems depend on choosing appropriate: Icons, graphical representations Natural and comprehensible data layouts

Example of DM: VisiCalc and Later … we’ve come a long way in 30 years … I guess People used to use paper and pencil for accounting, until 1980’s! Shneiderman –VisiCalc spreadsheet (1979) and its descendants –Imagine erasing a thousand numbers … “VisiCalc users delighted in watching the program propagate changes across the screen.” –In some cases, spatial representations provide a better model of reality –Successful spatial data- management systems depend on choosing appropriate: Icons, graphical representations Natural and comprehensible data layouts

Example of DM: WIMP File System Our paradigm example Copy, move, rename, delete –But, is it a file cabinet?

Example of DM: Spatial Data Mgmt Geographical data visualization and interaction Direct Manipulations –Spatial Data Management System (’80) Zoom-in on ocean map and marker buoys –Notion of using a joystick to navigate a map –BTW Idea: Nicholas Negroponte (MIT), 1970’s!

Example of DM: Spatial Data Mgmt … and spatial metaphors Others: –Xerox PARC Information Visualizer Lead article CACM, 1991 Implemented with high end work stations 3D file directories, org charts, 2d info Design is important –Icons, representations, understanding user need –Users usually enjoy the direct manipulation “Messing around”

Example of DM: Drawings Autocad, etc. Spatial

Example of DM: Statistical Analysis From Hutchins et al. - quaint Specify statistical operations by manipulating icons Similar to “visual programming”

Example of DM: Statistical Analysis From Hutchins et al. - quaint

Example of DM: Office Automation Xerox Star (1981) Apple Lisa (1983) –Precursor to Mac Direct manipulation –Drag file to printer –Pull-down menus –Window manipulation Microsoft Windows Command-line vs. GUI –Study result: task time (5.8 vs. 4.8 minutes), errors (2.0 vs. 0.8) (’87) –Subjectively preferred –novice/ computer naïve people really benefit –Improved productivity, reduced fatigue

Example of DM: CAD Computer Aided Design E.g., AutoCAD For structural engineer, landscaping, automobiles, etc. Change design and evaluate designs quickly Computer Aided Manufacturing (CAM) Allows many of specification tools to be used for large designs (group review, etc.) Few complex commands Analogy/familiar designs important (don’t change the terminology, etc.)

Example of DM: Games Verrrry successful application of DM Designing for different users –Age –Gender –Portability –Resolution/Computing Power –Genre –Multiplayer –Cultures Different controllers Note effect of having a score –public display, compare w/ friends, competition, better than encouragement Direct manipulation for education –SimCity –The Sims

Example of DM: Games Most successful application of DM … by far E.g., PONG –Low learning curve –Mass appeal (which many current games don’t have!)

Direct Manipulation The details … Conversational vs. direct interaction Shneiderman’s “high level” characterization – teased apart –User operates on visually displayed “world” to perform tasks in domain

Basic Interaction Paradigm In context of direct manipulation and command language Historically, “commands” differ by paradigm –Switches, cards –Command line –W-Icons-M-P Relatively, direct or indirect manipulation of system –“giving commands to system” Cmd lang: Conversational –User has a conversation with interface, telling it what to do –3 rd person –User tells system what to do in task Then system does it Direct Manipulation: Direct –No conversation User interacts directly with (representation) of task –1 st person –Operates directly on model worlds Task “work on task” “ gives commands” System User feedback “feedback” and its representation (possibly manipulable )

Stages of Action Models Overview More cognitively and task oriented description Norman's seven stages of action: 1.Forming the goal 2.Forming the intention 3.Specifying the action 4.Executing the action 5.Perceiving the system state 6.Interpreting the system state 7.Evaluating the outcome Some correspondences to Foley and van Dam levels of analysis –User forms a conceptual intention, reformulates it into the semantics of several commands, and eventually produces the lexical tokens Norman's contributions –Context of cycles of action and evaluation. –Gulf of execution: Mismatch between user's intentions and the allowable actions –Gulf of evaluation: Mismatch between system's representation and the users' expectations Following from this, Norman suggests four principles of good design: –…

General Comments about DM Schneiderman’s Historical Account … and Definition Shneiderman described in 1974 – will see sharper description Positive feelings associated with “good” user interfaces often attributed to “direct manipulation” interfaces: –Mastery of the interface –Competence in performing tasks –Ease in learning the system originally and in assimilating advanced features –Confidence in the capacity to retain mastery over time –Enjoyment in using the system –“Eagerness to show the system off to novices” <- ! –Desire to explore more powerful aspects of the system Definition: –User operates on visually displayed “world” to perform tasks in domain –All in all easier, to see examples, then analyze

Direct (and Indirect) Manipulation Interfaces “Direct manipulation” user interfaces contrasted with, –well, “indirect manipulation” interface Say, command line systems, or card systems, or switch systems, or … Recall, discussion of history of computing … relative costs, technology available But, that was a long time ago … Paradigm example: File system manipulation –Bottom line, file system in a computer is just a collection of storage elements orientations –Even idea of “files” is an abstraction –Organization of files in a hierarchy is a further abstraction Command line –cp /usr/fowler/presentations/colloquium_98/hypertextnodes.gif /usr/fowler/lectures/6362/dir_manip/ –Tell the system to take one file and make a copy of it and place that copy in the designated place –mv hypertextgraph.gif temp_hypertextgraph.gif –Tell the system to “move” one file to another file with a different name (rename)

Conversations and Model Worlds… In context of direct manipulation and command language Task “work on task” “ gives commands” System User feedback “feedback” and its representation (possibly manipulable ) Interface as conversation: –Interface is a language medium - user and computer have a conversation about an assumed, but not explicitly represented, world –Interface is intermediary between user and assumed world about which things are said (and in which commands are issued) Interface as model world: –Interface is itself a world where tuser can act and which changes state in response to user actions

Conversations and Model Worlds… In context of direct manipulation and command language Task “work on task” “ gives commands” System User feedback “feedback” and its representation (possibly manipulable ) Interface as conversation: –Interface is a language medium - user and computer have a conversation –Interface is intermediary Interface as model world: –Interface is itself a world where tuser can act and which changes state in response to user actions –World (domain) of interest is explicitly represented - there is no intermediary between user and world –Well done, this can create a sense of acting upon the object of the task domain themselves feeling of direct engagement

Direct Manipulation “Characterized” Again, … history and fyi An orientation –Really, direct manipulation is perhaps best characterized as an “orientation” to interface style “Definition/orientation”, yet again: –User operates on a visually displayed “world” to perform tasks in a domain Among the earliest characterizations (Schneiderman, 1982): 1. Continuous representation of the object of interest 2. Physical actions or labeled button presses instead of complex syntax 3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible Benefits (Schneiderman, 1982): 1. Novices can learn basic functionality quickly, often demonstration alone 2. Experts can work rapidly to carry out tasks, even defining new functions and features 3. Knowledgeable intermittent users can retain operational concepts 4. Error message are rarely needed (why?) 5. Users can see immediately if their actions are furthering their goals, and if not, they can simply change the direction of their activity

“Cognitive load”

Some Terminology Transduction of energy, etc. by sensory receptors –Light to neural impulses by retinal cells Sensation Perception Cognition Latin – cognitio – knowledge “Higher level” mental representations and procedures Process of thought Forming a “mental image” or awareness and representation To “see a window” Top down and bottom up

Model Human Processor + Attention Cognitive effort allocated at any number of points A “useful” big picture - Card et al. ’83 plus attention –Senses/input  f(attention, processing)  motor/output –Notion of “processors” Purely an engineering abstraction –More later Detail next slide

Model Human Processor + Attention Cognitive effort allocated at any number of points Sensory store –Rapid decay “buffer” to hold sensory, e.g., auditory, visual, input for later processing Perceptual processor –Recognizes symbols, phonemes –Aided by LTM Cognitive processor –Uses recognized symbols –Makes comparisons and decisions –Problem solving –Interacts with LTM and WM Motor processor –Input from cog. proc. for action –Instructs muscles –Feedback Results of muscles by senses Attention –Allocation of resources

Direct Manipulation “Characterized” Hutchins et al. orientation Typical characteristics of direct manipulation interfaces: 1. System (task) objects have visual representations 2. User interacts with a model world 3. No “conversation” about what to do the user directly manipulates the model world 4. Representations that match the user model facilitate feelings of directness 5. Output from one system action is input for another Hutchins et al.’s account –Delineates range of interface phenomena that can contribute to feeling of directness –Assumption is that the feeling of directness results from the commitment of fewer cognitive resources Or (put the other way around), the need to commit additional of the user’s cognitive resources leads to feelings of indirectness –And, so, account is in essence a cognitive account of direct manipulation interfaces

Hutchins et al. Paper Overview – later detail for all Provides a detailed analysis of the elements of direct manipulation Again, conversation metaphor and model world metaphor for interfaces Two aspects of directness in direct manipulation: –Distance and Engagement –directness = f(distance, engagement) Cognitive effort results from gulfs of execution and evaluation –Gulf = f (Mismatch of system elements and user’s task element) gulf of execution –match of commands and mechanisms of system with thoughts and goals of user gulf of evaluation –extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated Model worlds and inter-referential input/output Two forms of distance: Semantic and articulatory A space of interfaces: –Engagement x Distance from user goals

Two Aspects of Directness in Direct Manipulation 1. Distance and Engagement directness = f (distance, engagement) Directness –The feeling that results from interaction with an interface (direct vs. indirect) Distance –“Distance” between user’s thoughts and system’s requirements for using it to perform task (e.g., g32, s/edtor/editor - large distance) –Short distance = translation between user’s thoughts and system simple and straightforward (e.g., insert ‘i’, or type over “edtor” with “editor”) –Describes factors which underlie the generation of the feeling of directness

Two Aspects of Directness in Direct Manipulation Distance and 2. Engagement directness = f (distance, engagement) Engagement –Feeling that the user is directly manipulating the objects of interest e.g., dragging a subdirectory to another subdirectory vs. mv /usr/bob/*.* /usr/jo/ Again, conversation metaphor and model world metaphor for interfaces –talking about and describing an operation vs. –performing operation in a (visual) model world (representing the task domain)

directness = f(distance, engagement) Gulfs of Execution and Evaluation Distance: Relationship between user’s task and way task can be accomplished via interface –To enhance directness, need to minimize cognitive effort required to map interface actions to task actions i.e., “bridge the gulf of user’s goals and way they are specified in the system” “gulfs” exist to the extent of the mismatch of user’s task intentions and how system used to carry out those task intentions

directness = f(distance, engagement) Gulfs of Execution and Evaluation Distance: Relationship between user’s task and way task can be accomplished via interface –To enhance directness, need to minimize cognitive effort required to map interface actions to task actions i.e., “bridge the gulf of user’s goals and way they are specified in the system” “gulfs” exist to the extent of the mismatch of user’s task intentions and how system used to carry out those task intentions –Gulfs are about 1. making things happen and 2. knowing if happened –1. gulf of execution match of commands and mechanisms of system with thoughts and goals of user –2. gulf of evaluation extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated

Directness and Model World Metaphor 1 - Cognitive Effort, directness = f (1/cognitive effort) –Less effort, more directness –More effort, less directness –Leading to feelings of “direct engagement” with the task Cognitive effort results from gulf of execution and evaluation –Explains why systems are relatively easy or hard to use, and suggests how to minimize effort, or, make system easier to learn and use Task “work on task” “commands” System User “gives” “performs” “feedback” directness = (inversely related to, or 1/) amt of cognitive effort to manipulate system

Directness and Model World Metaphor 2 – Controlling world vs. interface Yet, “direct engagement” describes a qualitative feeling that user is directly engaged with control of objects (of task), rather than the computer or program which controls the task –e.g., really drawing the drawing vs. moving a cursor which, then, effects a line or fill Characterized as feeling of “first-personess” vs. “third- personess” –User operating on objects of task directly vs. telling someone (system) to operate on objects –Latter is surely case in “conversation metaphor” model of user interfaces User operates with linguistic structures (i.e., telling interface what to do) vs. User him/herself operating directly on the task objects (with no intermediary) *** Central metaphor of system is the model world itself - the key to direct manipulation interfaces

Model Worlds and Inter-referential Input/Output Operating in a model world, of course, requires that representations of task (model world) objects be manipulable in a direct, visual way Additionally, if user is to carry out a series of operations (i.e., effect a series of instructions/commands which change state of system) –Then necessary that results of one operation be available for further modification –That is, the output of one operation serve as input for another operation Term “inter-referential input/output” refers to this characteristic of interface objects –e.g., files and folders are moved and more moved again, etc. –words and paragraphs in editor undergo further modification (of course) Task “work on task” “commands” System User “gives” “performs” “feedback”

Two Forms of Distance: Semantic & Articulatory directness = f(distance, engagement) Term “language” is quite general –Refers in general to mapping (often arbitrary) of sign to meaning –Interface language – elements that cause change in system state –Two “dialects” Language for input (causing change to occur) Language for output (evaluating ensuing state of system), –e.g., in dm are files in new loc vs. result of mv... (in fact none unless cd and ls) Semantic Distance –Relationship of an expression in interface language to what user wants to say Articulatory Distance –Relationship between meaning of expression and its physical form Task “work on task” “commands” System User “gives” “performs” “feedback”

Two Forms of Distance: Semantic & Articulatory directness = f(distance, engagement) Term “language” is quite general –Refers in general to mapping (often arbitrary) of sign to meaning –Interface language – elements that cause change in system state –Two “dialects” Language for input (causing change to occur) Language for output (evaluating ensuing state of system), –e.g., in dm are files in new loc vs. result of mv... (in fact none unless cd and ls) –Semantic Distance Relationship of an expression in interface language to what user wants to say –Articulatory Distance Relationship between meaning of expression and its physical form

Two Forms of Distance: Semantic & Articulatory Gulf of Execution and Gulf of Evaluation … Cognitive Effort Extent to which there is “mismatch” between task and system, the “gulfs” become large –Gulf of Execution E.g., drag a file icon vs. “mv … /…/..”, “just want to move the _ file” Task goal <> system lang for cmd lang –Gulf of Evaluation Did it happen? “ls …./…./….” vs.

Two Forms of Distance: Semantic & Articulatory directness = f(distance, engagement) Semantic Distance –Relationship of expression in if. lang. to what user wants to say Articulatory Distance –Relationship between meaning of expression and its physical form –E.g., dragging a file icon to another location in a file structure– has both small semantic (goals) and articulatory (expression (=visual form) distances Because of metaphor Both relatively large for command langs Menu interface might have intermediate or less –Goals can be easy or hard to semantically map good vs. not menu structure –Likely, form of expression (~command) may have large dist. Task “work on task” “commands” System User “gives” “performs” “feedback”

Both Articulatory and Semantic Distances in Gulfs of Execution and Evaluation Relationships among: –semantic distance –articulatory distance and –gulfs of execution and evaluation Recall, menu example

Direct Engagement Summing up, “direct engagement” occurs when user experiences direct interactions with the objects in a domain –Again, “first-personness” no intermediary or conversation about what to do Model world represented in interface is the task –Interface becomes “invisible” or transparent or disappears –Not clear-cut how to create –And these feeling in fact can come from automization (learning) for many (or any?) interface Laurel (1986) suggests direct engagement requires: –Execution and evaluation should exhibit both semantic and articulatory directness –Input and output languages should be inter-referential allows to feel as if manipulating objects of concern –Rapid system response (ideally, 0 or time appropriate for task) –“Unobtrusive” interface interface is not to be noticed, if it is, leads to third-person relation

Direct Engagement Summing up, “direct engagement” occurs when user experiences direct interactions with the objects in a domain –Again, “first-personness” no intermediary or conversation about what to do Model world represented in interface is the task –Interface becomes “invisible” or transparent or disappears –Not clear-cut how to create –And these feeling in fact can come from automatization (learning) for many (or any?) interface –Laurel (1986) suggests direct engagement requires: Execution and evaluation should exhibit both semantic and articulatory directness Input and output languages should be inter-referential –allows to feel as if manipulating objects of concern Rapid system response (ideally, 0 or time appropriate for task) “Unobtrusive” interface –interface is not to be noticed, if it is, leads to third-person relation

A Space of Interfaces Direct manipulation interfaces minimize semantic and articulatory distances and maximize engagement

Problems with DM Interfaces Graphical interfaces, generally, have universal usability challenges –E.g., for visually impaired Uses much screen space for graphical representations of objects –Detail through text may be displaced May still have to learn meaning of visual representations –Not all visual representations have “straightforward” meaning Visual representation may be misleading –Even with analogy grasped, may still have to learn limitations Requires switching between keyboard and mouse Small screens with touch interfaces limit interaction

Challenges for DM Interfaces Choosing “right”, or useful, interface metaphor (that has a natural visual representation) a challenge … –What is a good metaphor for air traffic control? –What is the “best” metaphor –… and for whom E.g., file cabinet for secretaries, directed acyclic graph for computer scientists “Rapid (<100 ms), incremental, reversable” actions using graphic representations can have high processing demands, even by current standards –E.g., high processing load db systems –Doing this web-based currently a challenge

Summary – Principles of DM (from Shneiderman) Now we’ve the details! Principles: 1.Continuous representations of the objects and actions of interest with meaningful visual metaphors 2.Physical actions or presses of labeled buttons, instead of complex syntax 3.Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

Icons

.

Visual Thinking and Icons Exploiting visual nature of computers –Icon defined: An image, picture, or symbol representing a concept (or a small religious object) Visual Languages and Visual Thinking (Arnheim ’72) –Data viz and symbol people –Reaches out to the right-brained (look at all the users) –Shunned by many a left-brained Read a paper by an algorithm/theory person lately? WIMP interfaces have that nickname for a reason Individual cognitive styles vary, no one style –People think differently –Verbal vs. visual, Linear vs. holistic, “Left vs. right - brain –Should provide several if possible Depends on expected user base –Paint program (icons) vs. word processors (text menus) –When should you use it? –Ex. Road signs (left curve vs. bridge out vs stop). What factors play a part?

Icon Design Considerations Stand out from background and each other Limit the number 3D not necessarily good Familiarity (ex.) Selected icons should be easily found Animations, shadows, etc. help Dynamic icons –Size changes, thumbnails, etc. Interaction between icons

Icon Design Considerations Components of icons: –Lexical – brightness, color, blinking etc. –Syntatics – appearance and movements (lines, shape) –Semantics – object represented –Pragmatics – legibility, utility –Dynamics – receptivity to actions

Fyi - Visual Thinking and Icons Exploiting visual nature of computers Can challenge the first generation of hackers Individual cognitive styles vary: Verbal vs. visual Linear vs. holistic “Left vs. right - brain” Icon defined An image, picture, or symbol representing a concept (or a small religious object) Guidelines for icon design – Represent the object or action in a familiar manner – Limit the number of different icons – Make icons stand out from the background – Consider three-dimensional icons – Ensure a selected icon is visible from unselected icons – Design the movement animation – Add detailed information – Explore combinations of icons to create new objects or actions Five levels of icon design: – Lexical qualities – Syntactics – Semantics –Pragmatics – Dynamics

Fyi - Direct-Manipulation Programming Visual representations of information make direct-manipulation programming possible in other domains Example Demonstrational programming is when users create macros by simply doing their tasks The five challenges of programming in the user interface: 1. Sufficient computational generality 2. Access to the appropriate data structures and operators 3. Ease in programming and editing programs 4. Simplicity in invocation and assignment of arguments 5. Low risk Cognitive dimensions framework may help analyzing design issues of visual programming environments.

Direct Manipulation Programming “Visual languages” … in general

Example of DM: Statistical Analysis From Hutchins et al. - quaint Specify statistical operations by manipulating icons Similar to “visual programming”

Example of DM: Statistical Analysis From Hutchins et al. - quaint

Direct Manipulation Programming AVS (A visualization system)

Direct Manipulation Programming PITUI – programming in the user interface –Sufficient generality –Access to data structures and operators –Ease in programming and editing –Simplicity in execution and supplying arguments –Low-risk (low errors, reversibility, etc.) Cognitive-dimensions framework (Green and Petre ’96) –Analyzes design issues –Viscosity – difficulty in making changes –Progress evaluation – execute partial programs –Consistency, hidden dependences, visbility, etc.

Microsoft VPL Microsoft Visual Programming Language (VPL) is an application development environment designed on a graphical dataflow-based programming model. Rather than series of imperative commands sequentially executed, a dataflow program is more like a series of workers on an assembly line, who do their assigned task as the materials arrive. As a result VPL is well suited to programming a variety of concurrent or distributed processing scenarios.

Microsoft VPL A VPL dataflow consists of a connected sequence of activities represented as blocks with inputs and outputs that can be connected to other activity blocks. Figure 2 - Activity blocks have connections that represent messages sent from one activity to another

3D Interfaces We live in a 3D world Natural interfaces are better Therefore 3D interfaces will be the ultimate … or not –Natural interfaces aren’t always better! –Making the interface simple (thus unnatural) often aids performance Constrains movement Limiting possible actions –Depends on application and goal of the user interface Surgery simulation Military simulation (general vs. soldier training) Architecture, education, product design Video games

3D Interfaces Developments that show promise: –3D sound –Stereo display (Ware and Frank ’96) –Haptic feedback (mouse) 3D can help by: –Provide overviews to see big picture –Rapid teleportation (context shifts) –Zooming (aid disabled) –Multiple coordinated views (3dsmax) –3D icons can represent abstract or recognizable concepts What we really want are enhanced interfaces Give us powers we don’t normally have –Flying, x-ray vision, teleportation, etc.

Teleoperation Combines: –Direct Manipulation –Process Control Human operators control physical processes in complex environments –Mars rover control, flying airplanes (Predator), manufacturing, medicine (surgery) Supervisory control –Different levels of human control (automation) Direct Manipulation Issues –Adequate feedback (data quality, latency (transmission and operation delays), incomplete, interference) –Presence –Point and click or more natural interaction vs. typing

Virtual Reality and Immersive Interfaces Trying to simulate reality or an experience –Training, Learning, Exploring –Expensive –Logistically Difficult (physically) Immersive interfaces –Providing sensory input for full (or a high degree of) visual (and auditory) fields More later …

End