Download presentation
Presentation is loading. Please wait.
1
Interpretive Evaluation
Agenda Simple user modeling Interpretive Evaluation - Motivation - Methods Ethnography Fitt’s Law
2
Simpler User Modeling How do attributes of users influence the design of user interfaces? Are there some design guidelines that we can derive from different attributes? Fall 2002 CS/PSY 6750
3
User Profiles Attributes: Novice, intermediate, expert
attitude, motivation, reading level, typing skill, education, system experience, task experience, computer literacy, frequency of use, training, color-blindness, handedness, gender,… Novice, intermediate, expert Fall 2002 CS/PSY 6750
4
Motivation User Design goal Low motivation, discretionary use
Low motivation, mandatory High motivation, due to fear High motivation, due to interest Design goal Ease of learning Control, power Ease of learning, robustness, control Power, ease of use Fall 2002 CS/PSY 6750
5
Knowledge & Experience
task system low low high high low high high low Design goals Many syntactic and semantic prompts Efficient commands, concise syntax Semantic help facilities Lots of syntactic prompting Fall 2002 CS/PSY 6750
6
Job & Task Implications
Frequency of use High - Ease of use Low - Ease of learning & remembering Task implications Low - Ease of learning System use Mandatory - Ease of using Discretionary - Ease of learning Fall 2002 CS/PSY 6750
7
Evaluation Predictive Interpretive (today) User testing (later)
Discount Usability (Mon) Cognitive Modeling-GOMS family (Wed) Cognitive Modeling-contextual (next Mon) Interpretive (today) User testing (later) Fall 2002 CS/PSY 6750
8
Approaches Experimental (Lab studies, quantitative)
Typically in a closed, lab setting Manipulate independent variables to see effect on dependent variables Naturalistic (Field studies, qualitative) Observation occurs in “real life” setting Watch process over time “Ecologically valid” Fall 2002 CS/PSY 6750
9
Interpretive Evaluation
Experimental: Formal and objective Interpretive: More subjective Concerned with humans, so no objective reality Sociological, anthropological approach Users involved, as opposed to predictive approaches Fall 2002 CS/PSY 6750
10
Beliefs Sees limitations in scientific hypothesis testing in closed environment Lab is not real world Can’t control all variables Context is neglected Artificial, short tasks Fall 2002 CS/PSY 6750
11
IE Methods A number of different methods or techniques in this area exist Ethnography Our main focus Contextual inquiry More specific form of ethnography with a focus on asking questions Field study Common notion, often equated with ethnography Observational study Much more informal, just watching users … Fall 2002 CS/PSY 6750
12
Ethnography Deeply contextual study
Immerse oneself in situation you want to learn about (anthropological and sociological roots) Observing people in their cultural context Interpretation of data is primary Behavior is meaningful only in context Fall 2002 CS/PSY 6750
13
Philosophy Argues that formal environment of controlled study is artificial --- Experimenter wields “power” over subject So … get into working environment of user -> Interpretation is primary, rather than data Fall 2002 CS/PSY 6750
14
Objectives Understanding the user Understand goals and values
Understand individual’s or group’s interactions within a culture Try to make tacit domain knowledge explicit Do this in an unbiased fashion For UI designers: Improve system by finding problems in way it is currently being used Fall 2002 CS/PSY 6750
15
Techniques In person observation Audio/video recording Interviews
“Wallow in the data” Fall 2002 CS/PSY 6750
16
Observation is Key Carefully observe everything about users and their environment Think of describing it to someone who has never seen this activity before What users say is important, but also non-verbal details Fall 2002 CS/PSY 6750
17
Observations Things of interest to evaluator
Structure and language used in work Individual and group actions Culture affecting work Explicit and implicit aspects of work Example: Office work environment Business practices, rooms, artifacts, work standards, relationships between workers, managers, … Fall 2002 CS/PSY 6750
18
Interviews Important Have a question plan, but keep interview open to different directions Be specific Create interpretations together with users At end, query “What should I have asked?” Record interviews Fall 2002 CS/PSY 6750
19
Steps 1. Preparation 2. Field study
Understand organization policies and work culture Familiarize yourself with system and its history Set initial goals and prepare questions Gain access and permission to observe & interview 2. Field study Establish rapport with users Observe/interview users in workplace and collect all different forms of data Follow any leads that emerge from visits Record the visits Fall 2002 CS/PSY 6750 Rose et al ‘95
20
Steps 3. Analysis 4. Reporting
Compile collected data in numerical, textual and multimedia databases Quantify data and compile statistics Reduce and interpret data Refine goals and process used 4. Reporting Consider multiple audiences and goals Prepare a report and present findings Fall 2002 CS/PSY 6750
21
One Technique Affinity Diagram Process
Write down each quote/observation on a slip of paper Put up on board Coalesce items that have affinity If they are saying similar things about an issue Give names to different groups (colors too) Continue grouping subgroups A hierarchy will be formed Fall 2002 CS/PSY 6750
22
Why Useful? Can help designer gain a rich and true assessment of user needs Help to define requirements Uncovers true nature of user’s job Discovers things that are outside of job description or documentation Allows you to play role of end-user Can sit in when real users not available Open-ended and unbiased nature promotes discovery Empirical study and task analysis are more formal – ethnography may yield more unexpected revelations Fall 2002 CS/PSY 6750
23
Types of Findings Can be both Qualitative Quantitative
Observe trends, habits, patterns, … Quantitative How often was something done, what per cent of the time did something occur, how many different … Fall 2002 CS/PSY 6750
24
Drawbacks Time required Scale Type of results Acquired skill
Can take weeks or months Scale Most use small numbers of participants just to keep somewhat manageable Type of results Highly qualitative, may be difficult to present and use Acquired skill Identifying and extracting “interesting” things is challenging Fall 2002 CS/PSY 6750
25
Ethnomethodology Concurrent/informed ethnography
Study is being done in conjunction with a system being developed + Helps keep focus on user throughout design - Requires lots of time and coordination Fall 2002 CS/PSY 6750
26
Cooperative Evaluation
User is viewed as collaborator in evaluation, not a subject “Friendly approach” Relaxed version of think-aloud Evaluator and participant can ask each other questions Fall 2002 CS/PSY 6750
27
CE Methods Seeks to detect errors early in a prototype
Experimenter uses tasks, also talks to participant throughout, asks questions… Have debriefing session at end Fall 2002 CS/PSY 6750
28
Low Level Models Fitt’s Law
Models movement times for selection tasks Basic idea: Movement time for a well-rehearsed selection task Increases as the distance to the target increases Decreases as the size of the target increases Fall 2002 CS/PSY 6750
29
Components ID = log2 ( 2A/W ) ID - Index of difficulty distance
to move width tolerance of target bits result Both quantities are distances so unitless result Fall 2002 CS/PSY 6750
30
Components MT = b * ID or MT = a + b*ID MT - Movement time
MT is a linear function of ID Fall 2002 CS/PSY 6750
31
Original Experiment 1-D A W Fall 2002 CS/PSY 6750
32
Exact Equation Run empirical tests to determine a and b in MT = a + b*ID Will get different ones for different input devices and ways the device is used MT Fall 2002 ID CS/PSY 6750
33
Common Equation MT = a + b log2 (A/W + 1) Provides useful numbers
Fall 2002 CS/PSY 6750
34
Questions What do you do in 2D?
Where can this be applied in user interface design? Fall 2002 CS/PSY 6750
Similar presentations
