CSCW Evaluation Techniques Presented by: Christopher Edwards

Overview of Presentation Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion I have decided to keep this presentation tightly linked to the readings I assigned. The reasons for this are two fold: It provides an easy to follow framework for the presentation It allows a cognitive reference if people should have questions when conducting future evaluations of CSCW projects The broad overview of the presentation will contain these elements:

Evaluation Overview Olson and Olson. What are we Evaluating? Evaluations Techniques… Internal/External Validity Conclusion This segment contains: Using Olson and Olson paper Covering Concepts such as: Olson and Olson Conceptual framework Briefly review this Segment

Characteristics of Groups Individuals Differ in: Skills Ability Knowledge Personalities Motivations Agendas Members of Groups differ on: Skills --- Differing area of skills, diverse set of skills Ability----Knowing each other’s ability Knowledge--- Seasoned members of this lab, knowledge areas are diverse Personalities – can effect group relations and how individuals relate to the group. Motivations – MSc PhD all trying to build the knowledge of HCI. Agendas – differing directions for each member of the group – where you want to go? As group matures, roles status or power influence expectations and behavior Fit btwn Group Characteristics/Demand Characteristics ---which can Ultimately determine success/failure of collab tech. COORDINATOR failure in free-flowing group. Could be okay in structured group…

Characteristics of Organizations System comprised of people and technology Social Technology Physical Technology Comprised of multiple actors Dependency on Communication Information Processing Entities Represent key level in aggregation in CSCW System comprised of people Social Tech @ Org level includes: Rules, Routines, Procedures. Physical Tech – includes: Buildings, layout, furnishings and communic. Structure Org’s comprised of multiple actors, Actors have different capabilities, goals, agendas. Communication Dependency: Communication btwn Actors is Key Advancement in communication tech, means advancement of ORG’s CSCW tech’s are strong candidates for org’s. Also known as INFO Processing Entities – Technologies are a natural fit to facilitate this processing by shared information There are cognitive constructs used to characterize Org’s Also – Multiplicity of goal pursuit at diff levels of Org – reflected in Org culture

Characteristics of Task Tasks involve different types of material Physical, Digital or Ethereal Ease or Difficulty of Task Differ on Core Activity Subtasks – Tightly Coupled/Loosely coupled Fine grained approach to task by Olsonx2 Different types of materials: Physical – Chesters stuff Digital – Comp programming Etheral – Gestures Mikes stuff EASE/DIFF – depends on whether goal has large number of interconnected constraints or if it has familiar components Are there similar cases one can draw on? (Precursors to Collage Board?) Differ on Core Activity: which affects/ style of group members interaction Subtasks-- Given to individuals in Group Tight – One cannot begin or continue until another has given output Loose – Work can proceed in paralell Conflict/Cooperation of group can effect task

Characteristics of Environment Physical Environment Distance between Group members CSCW technologies designed to overcome Contextual Time When in the day the interaction occurs Effects on Distant Group member Physical environment – affects group work eg. lights, furniture, Distance of group members – closer people are more likely to collaborate CSCW tech’s are supposed to facilitation these interactions – trying to duplicate richness and ease of collab Contextual time – Collaboration over distance effected by time zones Holidays celebrations – affect local members, which may exclude distant collab’s

Characteristics of Technology Increasingly Varied Technologies to Support Conversation Auditory/Visual Back channels/gestures Technologies to Support Shared Work Objects that support work Fit of Tool to Material Technology to support group work is varied Face-Face has natural Using A/V ,gestures/backchannels/ turn taking Delay that tech’s impose – can uproot collab – Collage Board Tech support Work Objects help support the work process (Physical Objects) Can people collaborating tool effectively fit material of the work? Moving stages – design planning, organizing discussion

Process Technology Deployment How and Why Process Analysis Why outcomes were affected Progress of Task Communication process Examined through time scales result of tech deployment How tech was used How is changed what people do Group and Org’s behaviour reveal: Task progress: how time is spent Content of discussion Task allocation Measure of efficiency Comm. Process How much clarification is needed Turntaking – management of work flow What non-verbal communication occurs Social bonding Cooperation/Conflict Measuring participation Time scales: Unfolding group process over mins/hours Org process – days/weeks/years

Outcomes The initial outcome of using technology Quality of work Measure outcomes at every level Group Outcome Organizational Productivity For groups: Quality of work, judgment process or some characteristic of work Group Outcome How well they understand the product / or agree with it How satisfied they were about the process they engaged in ORG LEVEL: Productivity – learning, flexibility, innovation, customer satisfaction – economic indicators

Conceptual Framework for CSCW Studies Group Organization Process Outcomes Task Dynamic nature of life hinders research in CSCW Variations in: Groups, Org’s, tasks, technology and environment makes systematic comparisons difficult The framework developed by Olsonx2 It helps in the understanding of groupware – evaluation in groupware Key points (Group etc etc) – Process and Outcome Feedback loops – Using Tech in Process actually effects the use of that software Environment Technology

Tools used to evaluate CSCW technologies

The Survey Set of questions Fixed Alternatives Statistically Analyzed Wording of questions problematic Administered in large scales, usually written, also online, or over the phone. Fixed answers/alternative Stat analyzed – Likert scale Can effectively ask question No opportunity to expand answers

Interview Structure of interview Formal and structured Unstructured Analysis can be complicated Often involves more open ended queries Formal/ Structured – prep’d before hand asked in specific order. Unstructured/Informal– Allows for direction of conversation to focus on one aspect – more in depth Analysis can be complex – amount of material collected

Experimental Controlled Setting Specific Task Conditions Assignment of Participants Useful for making inferences about causality Controlled setting Participants assigned Quantitative analysis – complex Able to make inferences about causality Lab is different that field setting (VALIDITY)

Case Study Examines a single or small number of cases Exploratory research Examine a small # of cases in great depth Used as exploratory research – can reveal potential unexpected relationships LOTUS – analysis uncovered the importance of “fit” of system to Org’s incentive schemes

Ethnography Method adopted from Anthropology Describing Culture Used originally to describe other cultures Misunderstood method Comes from Anthro Methods applied to cultures Misunderstood – CSCW thinks that field observation is Ethnography Qualitative – narrative form

Many other Methods Diaries Analytic Field Studies Quasi Experimental Longitudinal Studies Historical Studies Diaries/Journals – accounts of activities by participant Analytic – Observational study, events are coded so that can be quantitatively analyzed Quasi – Pre-existing construct, field exp, can’t randomly assign participants Long – Traces Phenomena over time Historical Looking @ tech history – can reveal societal patterns….

Internal and External Validity HIGH Laboratory Experiments Field Experiments Level of Internal validity Surveys Methods differ on how valid they report phenomena Ext: Generalizability of findings in a setting, can be applicable to other situations or settings INT: Extent to which alternative explanations for behaviour can be eliminated There is some shifting of techniques – trying to simulate real world in a lab setting. Coding ethnographic data to be statistically analyzed Ethnographies LOW Level of External Validity HIGH

Conclusion to the Overview Different factors influence use and evaluation of CSCW software Framework of CSCW studies Evaluation Techniques Validity of Techniques Different elements influence the evaluation of technologies such as Groups, Org’s, Tasks, Environment, Tech how they react – process and outcomes Olsonx2 have provided a framework of those elements Vast amount of eval tools available and techniques. Pending on what you are looking for Validity of techniques can influence decision

Short Break Reconvene in 3 minutes *Upcoming – Understanding Ethnography Using Ethnography Ethnography and CSCW

Ethnography Understanding Ethnography Sociology Adoption Using Ethnography CSCW Ethnography in Design (Hughes) To use technique one must understand it Sociology adoption of Ethnographic tech Using Ethnography Ethnography in Design – examples etc

Understanding Ethnography Ethnography is loosely applied to qualitative research Home is originally from Anthropology Aim to describe cultural interpretation People commonly think that ethnography means Home in Anthro – to interpret distant cultures, webs of meanings in which they/we live

Understanding Ethnography Understanding culture “from an insiders point view” Three sources of data Participant Observation Interviews Collection of representative artifacts Using an inside view to understand and interpret culture How they do this is three ways. Engross themselves in the culture – part ob Interviews to elicit cultural knowledge In anthro – collection of artifacts that embody characteristics of the topic of interest. Using these data sources as a foundation – reliance on a cultural frame of analysis to infer tacit meanings shared by the group

Sociology Adoption Originally used to study distant cultures Chicago School of Sociology Studies focused on exploration of groups in urban settings Cultural comparisons in USA Family of Ethnographic Techniques Cultural anthro – ethno initially employed to describe the life patterns of people distant from their colonial homeland 1920’s Chicago School of Sociology adopted. Researched groups in urban settings – hobos, men on the street corner, alcholics Dependant on Focus in Socio – ethnography used for many areas. Cultural comparisons – compared to White male – looked at cultural discontinuity

Class Participation Time In Pairs (Saul and myself included) Everyone gets a Handout For a total of 5 Minutes (2.5 Minutes each) Each member of the pair (one at a time) asks the other questions from the sheet

Debrief of Class participation What answers were given…. Obviously not a long term ethnography study Depending on your relationship to this lab – differing perspectives Understanding Grouplab culture (to some extent) Through interview – trying to grasp cultural influences of grouplab, culture specific?

Ethnography and CSCW Prominence of Ethnography in CSCW Insufficient attention to social context New problems for design of collaborative character of work and activities Ethnography and system design Problem of scale Pressure of time Role of the ethnographer Response to sociality of work Comp Moving into world of work and organizations Drift to sociology methods – social nature of work Concerned to produce detailed descriptions of the workaday activities of social actors in specific circumstances Portrait of life Social actions embedded within a organized domain Understand a social setting from the point of view of those in the setting Assimilation of ethno results problematic Discursive format – rambling Design options not clearly stated Decontextualise – find processes in social setting – groups working together

Concurrent Ethnography Design is influenced by on-going ethnographic study Sequenced process Ethnographic Study Ethnographic study at the same time as systems development Systems Development Debriefing Meetings System Prototype

Concurrent Ethnography in action London Air Traffic Control Centre Four week Ethnography Each stage of fieldwork was intended to target designers issues Small research team What ethnography provided Research on LATCC Four week concurrent ethnography Initial phenomena

Quick and Dirty Ethnography Brief Ethnographic Studies Duration relative to the size of the task Selecting aspects of work setting of importance to design Outline of project Meetings Valuable knowledge of social organization of work of a relatively large scale work setting in a short space of time. Short Focus Studies Debriefing Meetings Scoping Document

Quick and Dirty Ethnography in Action Ethnographic investigation of software engineers Challenges of Large scale setting Working in Industrialized Environments Acceptance into the setting (*Key to Ethnographic research) To inform the design of a support tool Aim to develop tool that reflected real world of design world.

Evaluative Ethnography Ethnography used to verify formulated design decisions Initial outline Design or Specification Short Ethnographic study Debriefing Meetings Amended Design Or Specification

Evaluative Ethnography in action Fieldwork in Building Society Using research for IT developments Routine of work Finding what customers wanted Outlined limitations of model that had been proposed

Re-examination of previous studies Previous studies are re-examined to inform design Ethnography used for many decades Many studies related to work and occupation Can be informative

Re-examination in action Inform preliminary design of Shared Object Service Using previous Ethnographic studies on: Social work, police work and invoice processing in a multi-site fast food company What common service should support

Summary of Ethnography Understanding Ethnography Ethnography and CSCW Uses of Ethnography Concurrent Quick and Dirty Evaluative Re-examination

Big Break Time 5 Minutes Reconvene in 5 Minutes Upcoming – Ethnomethodology Understanding Ethnomethodology Ethnomethodology in CSCW

Ethnomethodology Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology in system design Incorporation of Sociology and Computer Science? ‘Technomethodology’

Understanding Ethnomethodology Ethnomethodology literally means “People’s Methods” A Shift from ‘other’ Sociological Methods Social Life is potentially Chaotic Social Actors Members methods for making sense Social structure exists. Impose sociological categories on the ordinary person Critique of Mainstream Sociology, rebelling to social structure. How members make sense of impressions/experiences. Organize them into a pattern…

Understanding Ethnomethodology Garfinkel “Documentary Method” Example of Documentary Method Garfinkel “Indexicality” Disrupt Technique Example in class Doc Method: Individuals make sense of their social world through psyc processes. Selecting fasts from a certain social situation, which conforms to a pattern, making sense of pattern. Once pattern established – used as framework – which can help interpret new situations. New Form of Psychotherapy – Advisor – Random Numbers – Yes/No – Should I quit school – No consistency in answers. Students made sense of “Advice Given” Garfinkel – students constructing social reality – to make sense of the senseless Indexicality – Making sense of REMARK, ACTION, SIGN, by reference of context they are in.

Understanding Ethnomethodology We can observe other members methods of construction Development of Conversation Analysis Members construct social action – we can observe this construction. CA – Conversation as social action, rather than articulation of mental states. How aspects of conversation are MANAGED – practical activity. Natural language – provide order and management of social settings. Record conversations – Audio/Video – capturing information (Ethnography)

Confusing Ethnography and Ethnomethodology Ethnography is a form of investigative fieldwork Ethnography focuses on the “Member’s Point of View” Ethnomethodology is a specific analytical technique Gathering data Perspective issue Gathering work group information

Confusing Ethnography and Ethnomethodology Confusion arises because: Ethnomethodologist is likely to use ethnographic techniques ‘Analytic mentality’-selection of phenomena and topics for investigation How people make sense of the world they are in.

Ethnomethodology in HCI and CSCW Observations of work activities and interactions help design process Understanding temporal organization of activities and interactions and implications to design It helps how people construct their social situation – pattern forming, which can help with design. Ethnomethodology focuses on what everyday accomplishment of work – not abstract models of activity.

Learning from Ethnomethodologists Division of Labour Field Observation conducted by ethnomethodologists Act as proxy for end users Hand off requirements to computer science people

Ethnomethodology for Critique and Design Ethnomethodology has provided: Critique of the design Failure to support the work Technology doesn’t allow people to engage in their work Outlines organization of work and communication in the real world Everyday accomplishment of work.

Two Paradoxes Paradox of system design Paradox of technomethodology Large scale activity Paradox of technomethodology Transformational nature of technology Analysis of practice not invention Trying to fit large scale work – into a small scale working (detailed) Trying to fit this technomethodology to CSCW area….iffy

Technomethodology Develop a stance in which ethnomethodology and computer science play equally significant roles Foundational relationships Concerns foundational concepts of system design, not process by which system design proceeds. Formulating foundational relationships on which to proceed.

Technomethodology Trying to exploit generalizations from ethnomethodology Abstractions from both disciplines Means by which such working practices arise Dialogical interfaces Exploit generalities not in terms of particular application setting but in terms of conceptual underpinning of system design – can they really do this?. CPSC – Abstractions are used to create behavior-really? Ethnometh – Abstractions are used to characterize and explain behavior. Technometh avoids specific working practices – focuses on means by which such working practices arise and are constituted. Turn taking model – how conversational flow is sustained, rather than by which they might be enacted. Management of conversation – not specifics of human dialogue.

Conclusion Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology and system design Technomethodology

Conclusion of Presentation Overview of Evaluation Techniques used in CSCW Ethnography Ethnomethodology Technomethodology

My thoughts CSCW has been unfortunately caught in a battle between sociological techniques CSCW design should incorporate work context issues

It was this clear….