CSCW Evaluation Techniques Presented by: Christopher Edwards

Overview of Presentation Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion

Evaluation Overview Olson and Olson. What are we Evaluating? Evaluations Techniques… Internal/External Validity Conclusion

Characteristics of Groups Individuals Differ in: Skills Ability Knowledge Personalities Motivations Agendas

Characteristics of Organizations System comprised of people and technology Social Technology Physical Technology Comprised of multiple actors Dependency on Communication Information Processing Entities

Characteristics of Task Tasks involve different types of material Physical, Digital or Ethereal Ease or Difficulty of Task Differ on Core Activity Subtasks – Tightly Coupled/Loosely coupled

Characteristics of Environment Physical Environment Distance between Group members CSCW technologies designed to overcome Contextual Time When in the day the interaction occurs Effects on Distant Group member

Characteristics of Technology Increasingly Varied Technologies to Support Conversation Auditory/Visual Back channels/gestures Technologies to Support Shared Work Objects that support work Fit of Tool to Material

Process Technology Deployment How and Why Process Analysis Why outcomes were affected Progress of Task Communication process Examined through time scales

Outcomes The initial outcome of using technology Quality of work Measure outcomes at every level Group Outcome Organizational Productivity

Conceptual Framework for CSCW Studies Group Organization Task Environment Technology ProcessOutcomes

Tools used to evaluate CSCW technologies

The Survey Set of questions Fixed Alternatives Statistically Analyzed Wording of questions problematic

Interview Structure of interview Formal and structured Unstructured Analysis can be complicated

Experimental Controlled Setting Specific Task Conditions Assignment of Participants Useful for making inferences about causality

Case Study Examines a single or small number of cases Exploratory research

Ethnography Method adopted from Anthropology Describing Culture Used originally to describe other cultures Misunderstood method

Many other Methods Diaries Analytic Field Studies Quasi Experimental Longitudinal Studies Historical Studies

Internal and External Validity HIGH LOW HIGH Level of Internal validity Level of External Validity Laboratory Experiments Surveys Field Experiments Ethnographies

Conclusion to the Overview Different factors influence use and evaluation of CSCW software Framework of CSCW studies Evaluation Techniques Validity of Techniques

Short Break Reconvene in 3 minutes *Upcoming – Understanding Ethnography Using Ethnography Ethnography and CSCW

Ethnography Understanding Ethnography Sociology Adoption Using Ethnography CSCW Ethnography in Design (Hughes)

Understanding Ethnography Ethnography is loosely applied to qualitative research Home is originally from Anthropology Aim to describe cultural interpretation

Understanding Ethnography Understanding culture from an insiders point view Three sources of data Participant Observation Interviews Collection of representative artifacts

Sociology Adoption Originally used to study distant cultures Chicago School of Sociology Studies focused on exploration of groups in urban settings Cultural comparisons in USA Family of Ethnographic Techniques

Class Participation Time In Pairs (Saul and myself included) Everyone gets a Handout For a total of 5 Minutes (2.5 Minutes each) Each member of the pair (one at a time) asks the other questions from the sheet

Debrief of Class participation What answers were given…. Obviously not a long term ethnography study Depending on your relationship to this lab – differing perspectives Understanding Grouplab culture (to some extent)

Ethnography and CSCW Prominence of Ethnography in CSCW Insufficient attention to social context New problems for design of collaborative character of work and activities Ethnography and system design Problem of scale Pressure of time Role of the ethnographer

Concurrent Ethnography Design is influenced by on-going ethnographic study Sequenced process Debriefing Meetings System Prototype Ethnographic Study Systems Development

Concurrent Ethnography in action London Air Traffic Control Centre Four week Ethnography Each stage of fieldwork was intended to target designers issues Small research team What ethnography provided

Quick and Dirty Ethnography Brief Ethnographic Studies Duration relative to the size of the task Selecting aspects of work setting of importance to design Outline of project Meetings Debriefing Meetings Scoping Document Short Focus Studies

Quick and Dirty Ethnography in Action Ethnographic investigation of software engineers Challenges of Large scale setting Working in Industrialized Environments Acceptance into the setting (*Key to Ethnographic research)

Evaluative Ethnography Ethnography used to verify formulated design decisions Short Ethnographic study Initial outline Design or Specification Debriefing Meetings Amended Design Or Specification

Evaluative Ethnography in action Fieldwork in Building Society Using research for IT developments Routine of work Finding what customers wanted Outlined limitations of model that had been proposed

Re-examination of previous studies Previous studies are re-examined to inform design Ethnography used for many decades Many studies related to work and occupation Can be informative

Re-examination in action Inform preliminary design of Shared Object Service Using previous Ethnographic studies on: Social work, police work and invoice processing in a multi-site fast food company What common service should support

Summary of Ethnography Understanding Ethnography Ethnography and CSCW Uses of Ethnography Concurrent Quick and Dirty Evaluative Re-examination

Big Break Time 5 Minutes Reconvene in 5 Minutes Upcoming – Ethnomethodology Understanding Ethnomethodology Ethnomethodology in CSCW

Ethnomethodology Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology in system design Incorporation of Sociology and Computer Science? Technomethodology

Understanding Ethnomethodology Ethnomethodology literally means Peoples Methods A Shift from other Sociological Methods Social Life is potentially Chaotic Social Actors Members methods for making sense

Understanding Ethnomethodology Garfinkel Documentary Method Example of Documentary Method Garfinkel Indexicality Disrupt Technique Example in class

Understanding Ethnomethodology We can observe other members methods of construction Development of Conversation Analysis

Confusing Ethnography and Ethnomethodology Ethnography is a form of investigative fieldwork Ethnography focuses on the Members Point of View Ethnomethodology is a specific analytical technique

Confusing Ethnography and Ethnomethodology Confusion arises because: Ethnomethodologist is likely to use ethnographic techniques Analytic mentality-selection of phenomena and topics for investigation

Ethnomethodology in HCI and CSCW Observations of work activities and interactions help design process Understanding temporal organization of activities and interactions and implications to design

Learning from Ethnomethodologists Division of Labour Field Observation conducted by ethnomethodologists Act as proxy for end users Hand off requirements to computer science people

Ethnomethodology for Critique and Design Ethnomethodology has provided: Critique of the design Failure to support the work Technology doesnt allow people to engage in their work Outlines organization of work and communication in the real world

Two Paradoxes Paradox of system design Large scale activity Paradox of technomethodology Transformational nature of technology Analysis of practice not invention

Technomethodology Develop a stance in which ethnomethodology and computer science play equally significant roles Foundational relationships

Technomethodology Trying to exploit generalizations from ethnomethodology Abstractions from both disciplines Means by which such working practices arise Dialogical interfaces

Conclusion Understanding Ethnomethodology Confusing Ethnography and Ethnomethodology Ethnomethodology and system design Technomethodology

Conclusion of Presentation Overview of Evaluation Techniques used in CSCW Ethnography Ethnomethodology Technomethodology

My thoughts CSCW has been unfortunately caught in a battle between sociological techniques CSCW design should incorporate work context issues

It was this clear….