1. CSCW Evaluation Techniques
Presented by:
Christopher Edwards

2. Overview of Presentation
Evaluation Techniques
Understanding Ethnography
Using Ethnography in CSCW
Understanding Ethnomethodology
Ethnomethodology and CSCW
Technomethodology
Conclusion
I have decided to keep this presentation tightly linked to the readings I assigned.
The reasons for this are two fold:
It provides an easy to follow framework for the presentation
It allows a cognitive reference if people should have questions when conducting future evaluations of CSCW projects
The broad overview of the presentation will contain these elements:

3. Evaluation Overview Olson and Olson. What are we Evaluating?
Evaluations Techniques…
Internal/External Validity
Conclusion
This segment contains:
Using Olson and Olson paper
Covering Concepts such as:
Olson and Olson Conceptual framework
Briefly review this Segment

4. Characteristics of Groups
Individuals Differ in:
Skills
Ability
Knowledge
Personalities
Motivations
Agendas
Members of Groups differ on:
Skills --- Differing area of skills, diverse set of skills
Ability----Knowing each other’s ability
Knowledge--- Seasoned members of this lab, knowledge areas are diverse
Personalities – can effect group relations and how individuals relate to the group.
Motivations – MSc PhD all trying to build the knowledge of HCI.
Agendas – differing directions for each member of the group – where you want to go?
As group matures, roles status or power influence expectations and behavior
Fit btwn Group Characteristics/Demand Characteristics ---which can Ultimately determine success/failure of collab tech.
COORDINATOR failure in free-flowing group. Could be okay in structured group…

5. Characteristics of Organizations
System comprised of people and technology
Social Technology
Physical Technology
Comprised of multiple actors
Dependency on Communication
Information Processing Entities
Represent key level in aggregation in CSCW
System comprised of people
Social Org level includes: Rules, Routines, Procedures.
Physical Tech – includes: Buildings, layout, furnishings and communic. Structure
Org’s comprised of multiple actors,
Actors have different capabilities, goals, agendas.
Communication Dependency: Communication btwn Actors is Key
Advancement in communication tech, means advancement of ORG’s
CSCW tech’s are strong candidates for org’s.
Also known as INFO Processing Entities – Technologies are a natural fit to facilitate this processing by shared information
There are cognitive constructs used to characterize Org’s
Also – Multiplicity of goal pursuit at diff levels of Org – reflected in Org culture

6. Characteristics of Task
Tasks involve different types of material
Physical, Digital or Ethereal
Ease or Difficulty of Task
Differ on Core Activity
Subtasks – Tightly Coupled/Loosely coupled
Fine grained approach to task by Olsonx2
Different types of materials:
Physical – Chesters stuff
Digital – Comp programming
Etheral – Gestures Mikes stuff
EASE/DIFF – depends on whether goal has large number of interconnected constraints or if it has familiar components
Are there similar cases one can draw on? (Precursors to Collage Board?)
Differ on Core Activity: which affects/ style of group members interaction
Subtasks-- Given to individuals in Group
Tight – One cannot begin or continue until another has given output
Loose – Work can proceed in paralell
Conflict/Cooperation of group can effect task

7. Characteristics of Environment
Physical Environment
Distance between Group members
CSCW technologies designed to overcome
Contextual Time
When in the day the interaction occurs
Effects on Distant Group member
Physical environment – affects group work eg. lights, furniture,
Distance of group members – closer people are more likely to collaborate
CSCW tech’s are supposed to facilitation these interactions – trying to duplicate richness and ease of collab
Contextual time – Collaboration over distance effected by time zones
Holidays celebrations – affect local members, which may exclude distant collab’s

8. Characteristics of Technology
Increasingly Varied
Technologies to Support Conversation
Auditory/Visual
Back channels/gestures
Technologies to Support Shared Work
Objects that support work
Fit of Tool to Material
Technology to support group work is varied
Face-Face has natural
Using A/V ,gestures/backchannels/ turn taking
Delay that tech’s impose – can uproot collab – Collage Board
Tech support Work
Objects help support the work process (Physical Objects)
Can people collaborating tool effectively fit material of the work?
Moving stages – design planning, organizing discussion

9. Process Technology Deployment How and Why Process Analysis
Why outcomes were affected
Progress of Task
Communication process
Examined through time scales
result of tech deployment
How tech was used
How is changed what people do
Group and Org’s behaviour reveal:
Task progress: how time is spent
Content of discussion
Task allocation
Measure of efficiency
Comm. Process
How much clarification is needed
Turntaking – management of work flow
What non-verbal communication occurs
Social bonding
Cooperation/Conflict
Measuring participation
Time scales:
Unfolding group process over mins/hours
Org process – days/weeks/years

10. Outcomes The initial outcome of using technology Quality of work
Measure outcomes at every level
Group Outcome
Organizational Productivity
For groups:
Quality of work, judgment process or some characteristic of work
Group Outcome
How well they understand the product / or agree with it
How satisfied they were about the process they engaged in
ORG LEVEL:
Productivity – learning, flexibility, innovation, customer satisfaction – economic indicators

11. Conceptual Framework for CSCW Studies
Group
Organization
Process
Outcomes
Task
Dynamic nature of life hinders research in CSCW
Variations in: Groups, Org’s, tasks, technology and environment makes systematic comparisons difficult
The framework developed by Olsonx2
It helps in the understanding of groupware – evaluation in groupware
Key points (Group etc etc) – Process and Outcome
Feedback loops – Using Tech in Process actually effects the use of that software
Environment
Technology

12. Tools used to evaluate CSCW technologies

13. The Survey Set of questions Fixed Alternatives Statistically Analyzed
Wording of questions problematic
Administered in large scales, usually written, also online, or over the phone.
Fixed answers/alternative
Stat analyzed – Likert scale
Can effectively ask question
No opportunity to expand answers

14. Interview Structure of interview Formal and structured Unstructured
Analysis can be complicated
Often involves more open ended queries
Formal/ Structured – prep’d before hand asked in specific order.
Unstructured/Informal–
Allows for direction of conversation to focus on one aspect – more in depth
Analysis can be complex – amount of material collected

15. Experimental Controlled Setting Specific Task Conditions
Assignment of Participants
Useful for making inferences about causality
Controlled setting
Participants assigned
Quantitative analysis – complex
Able to make inferences about causality
Lab is different that field setting (VALIDITY)

16. Case Study Examines a single or small number of cases
Exploratory research
Examine a small # of cases in great depth
Used as exploratory research – can reveal potential unexpected relationships
LOTUS – analysis uncovered the importance of “fit” of system to Org’s incentive schemes

17. Ethnography Method adopted from Anthropology Describing Culture
Used originally to describe other cultures
Misunderstood method
Comes from Anthro
Methods applied to cultures
Misunderstood – CSCW thinks that field observation is Ethnography
Qualitative – narrative form

18. Many other Methods Diaries Analytic Field Studies Quasi Experimental
Longitudinal Studies
Historical Studies
Diaries/Journals – accounts of activities by participant
Analytic – Observational study, events are coded so that can be quantitatively analyzed
Quasi – Pre-existing construct, field exp, can’t randomly assign participants
Long – Traces Phenomena over time
Historical tech history – can reveal societal patterns….

19. Internal and External Validity
HIGH
Laboratory
Experiments
Field
Experiments
Level of
Internal
validity
Surveys
Methods differ on how valid they report phenomena
Ext: Generalizability of findings in a setting, can be applicable to other situations or settings
INT: Extent to which alternative explanations for behaviour can be eliminated
There is some shifting of techniques – trying to simulate real world in a lab setting. Coding ethnographic data to be statistically analyzed
Ethnographies
LOW
Level of External Validity
HIGH

20. Conclusion to the Overview
Different factors influence use and evaluation of CSCW software
Framework of CSCW studies
Evaluation Techniques
Validity of Techniques
Different elements influence the evaluation of technologies such as
Groups, Org’s, Tasks, Environment, Tech how they react – process and outcomes
Olsonx2 have provided a framework of those elements
Vast amount of eval tools available and techniques.
Pending on what you are looking for Validity of techniques can influence decision

21. Short Break Reconvene in 3 minutes
*Upcoming – Understanding Ethnography
Using Ethnography
Ethnography and CSCW

22. Ethnography Understanding Ethnography Sociology Adoption
Using Ethnography
CSCW Ethnography in Design (Hughes)
To use technique one must understand it
Sociology adoption of Ethnographic tech
Using Ethnography
Ethnography in Design – examples etc

23. Understanding Ethnography
Ethnography is loosely applied to qualitative research
Home is originally from Anthropology
Aim to describe cultural interpretation
People commonly think that ethnography means
Home in Anthro – to interpret distant cultures, webs of meanings in which they/we live

24. Understanding Ethnography
Understanding culture “from an insiders point view”
Three sources of data
Participant Observation
Interviews
Collection of representative artifacts
Using an inside view to understand and interpret culture
How they do this is three ways.
Engross themselves in the culture – part ob
Interviews to elicit cultural knowledge
In anthro – collection of artifacts that embody characteristics of the topic of interest.
Using these data sources as a foundation – reliance on a cultural frame of analysis to infer tacit meanings shared by the group

25. Sociology Adoption Originally used to study distant cultures
Chicago School of Sociology
Studies focused on exploration of groups in urban settings
Cultural comparisons in USA
Family of Ethnographic Techniques
Cultural anthro – ethno initially employed to describe the life patterns of people distant from their colonial homeland
1920’s Chicago School of Sociology adopted.
Researched groups in urban settings – hobos, men on the street corner, alcholics
Dependant on Focus in Socio – ethnography used for many areas.
Cultural comparisons – compared to White male – looked at cultural discontinuity

26. Class Participation Time
In Pairs (Saul and myself included)
Everyone gets a Handout
For a total of 5 Minutes (2.5 Minutes each)
Each member of the pair (one at a time) asks the other questions from the sheet

27. Debrief of Class participation
What answers were given….
Obviously not a long term ethnography study
Depending on your relationship to this lab – differing perspectives
Understanding Grouplab culture (to some extent)
Through interview – trying to grasp cultural influences of grouplab, culture specific?

28. Ethnography and CSCW Prominence of Ethnography in CSCW
Insufficient attention to social context
New problems for design of collaborative character of work and activities
Ethnography and system design
Problem of scale
Pressure of time
Role of the ethnographer
Response to sociality of work
Comp Moving into world of work and organizations
Drift to sociology methods – social nature of work
Concerned to produce detailed descriptions of the workaday activities of social actors in specific circumstances
Portrait of life
Social actions embedded within a organized domain
Understand a social setting from the point of view of those in the setting
Assimilation of ethno results problematic
Discursive format – rambling
Design options not clearly stated
Decontextualise – find processes in social setting – groups working together

29. Concurrent Ethnography
Design is influenced by on-going ethnographic study
Sequenced process
Ethnographic
Study
Ethnographic study at the same time as systems development
Systems
Development
Debriefing Meetings
System Prototype

30. Concurrent Ethnography in action
London Air Traffic Control Centre
Four week Ethnography
Each stage of fieldwork was intended to target designers issues
Small research team
What ethnography provided
Research on LATCC
Four week concurrent ethnography
Initial phenomena

31. Quick and Dirty Ethnography
Brief Ethnographic Studies
Duration relative to the size of the task
Selecting aspects of work setting of importance to design
Outline of project
Meetings
Valuable knowledge of social organization of work of a relatively large scale work setting in a short space of time.
Short
Focus
Studies
Debriefing Meetings
Scoping Document

32. Quick and Dirty Ethnography in Action
Ethnographic investigation of software engineers
Challenges of Large scale setting
Working in Industrialized Environments
Acceptance into the setting (*Key to Ethnographic research)
To inform the design of a support tool
Aim to develop tool that reflected real world of design world.

33. Evaluative Ethnography
Ethnography used to verify formulated design decisions
Initial outline
Design or
Specification
Short
Ethnographic
study
Debriefing Meetings
Amended Design
Or Specification

34. Evaluative Ethnography in action
Fieldwork in Building Society
Using research for IT developments
Routine of work
Finding what customers wanted
Outlined limitations of model that had been proposed

35. Re-examination of previous studies
Previous studies are re-examined to inform design
Ethnography used for many decades
Many studies related to work and occupation
Can be informative

36. Re-examination in action
Inform preliminary design of Shared Object Service
Using previous Ethnographic studies on:
Social work, police work and invoice processing in a multi-site fast food company
What common service should support

37. Summary of Ethnography
Understanding Ethnography
Ethnography and CSCW
Uses of Ethnography
Concurrent
Quick and Dirty
Evaluative
Re-examination

38. Big Break Time 5 Minutes Reconvene in 5 Minutes
Upcoming – Ethnomethodology
Understanding Ethnomethodology
Ethnomethodology in CSCW

39. Ethnomethodology Understanding Ethnomethodology
Confusing Ethnography and Ethnomethodology
Ethnomethodology in system design
Incorporation of Sociology and Computer Science? ‘Technomethodology’

40. Understanding Ethnomethodology
Ethnomethodology literally means “People’s Methods”
A Shift from ‘other’ Sociological Methods
Social Life is potentially Chaotic
Social Actors
Members methods for making sense
Social structure exists. Impose sociological categories on the ordinary person
Critique of Mainstream Sociology, rebelling to social structure.
How members make sense of impressions/experiences. Organize them into a pattern…

41. Understanding Ethnomethodology
Garfinkel “Documentary Method”
Example of Documentary Method
Garfinkel “Indexicality”
Disrupt Technique
Example in class
Doc Method: Individuals make sense of their social world through psyc processes.
Selecting fasts from a certain social situation, which conforms to a pattern, making sense of pattern.
Once pattern established – used as framework – which can help interpret new situations.
New Form of Psychotherapy – Advisor – Random Numbers – Yes/No – Should I quit school – No consistency in answers.
Students made sense of “Advice Given”
Garfinkel – students constructing social reality – to make sense of the senseless
Indexicality – Making sense of REMARK, ACTION, SIGN, by reference of context they are in.

42. Understanding Ethnomethodology
We can observe other members methods of construction
Development of Conversation Analysis
Members construct social action – we can observe this construction.
CA –
Conversation as social action, rather than articulation of mental states.
How aspects of conversation are MANAGED – practical activity.
Natural language – provide order and management of social settings.
Record conversations – Audio/Video – capturing information (Ethnography)

43. Confusing Ethnography and Ethnomethodology
Ethnography is a form of investigative fieldwork
Ethnography focuses on the “Member’s Point of View”
Ethnomethodology is a specific analytical technique
Gathering data
Perspective issue
Gathering work group information

44. Confusing Ethnography and Ethnomethodology
Confusion arises because:
Ethnomethodologist is likely to use ethnographic techniques
‘Analytic mentality’-selection of phenomena and topics for investigation
How people make sense of the world they are in.

45. Ethnomethodology in HCI and CSCW
Observations of work activities and interactions help design process
Understanding temporal organization of activities and interactions and implications to design
It helps how people construct their social situation
– pattern forming, which can help with design.
Ethnomethodology focuses on what everyday accomplishment of work
– not abstract models of activity.

46. Learning from Ethnomethodologists
Division of Labour
Field Observation conducted by ethnomethodologists
Act as proxy for end users
Hand off requirements to computer science people

47. Ethnomethodology for Critique and Design
Ethnomethodology has provided:
Critique of the design
Failure to support the work
Technology doesn’t allow people to engage in their work
Outlines organization of work and communication in the real world
Everyday accomplishment of work.

48. Two Paradoxes Paradox of system design Paradox of technomethodology
Large scale activity
Paradox of technomethodology
Transformational nature of technology
Analysis of practice not invention
Trying to fit large scale work – into a small scale working (detailed)
Trying to fit this technomethodology to CSCW area….iffy

49. Technomethodology
Develop a stance in which ethnomethodology and computer science play equally significant roles
Foundational relationships
Concerns foundational concepts of system design, not process by which system design proceeds.
Formulating foundational relationships on which to proceed.

50. Technomethodology
Trying to exploit generalizations from ethnomethodology
Abstractions from both disciplines
Means by which such working practices arise
Dialogical interfaces
Exploit generalities not in terms of particular application setting but in terms of conceptual underpinning of system design – can they really do this?.
CPSC – Abstractions are used to create behavior-really?
Ethnometh – Abstractions are used to characterize and explain behavior.
Technometh avoids specific working practices – focuses on means by which such working practices arise and are constituted.
Turn taking model – how conversational flow is sustained, rather than by which they might be enacted. Management of conversation – not specifics of human dialogue.

51. Conclusion Understanding Ethnomethodology
Confusing Ethnography and Ethnomethodology
Ethnomethodology and system design
Technomethodology

52. Conclusion of Presentation
Overview of Evaluation Techniques used in CSCW
Ethnography
Ethnomethodology
Technomethodology

53. My thoughts
CSCW has been unfortunately caught in a battle between sociological techniques
CSCW design should incorporate work context issues

54. It was this clear….