© deSouza, Dittrich, Sharp Doing Ethnography

© deSouza, Dittrich, Sharp Key principles Everything is strange –Question why? –Stop and reflect Members’ point of view –Developer themselves – why do that? No a priori expectations –Non judgmental –Data analysis and changing plans continuous

© deSouza, Dittrich, Sharp Before the study Recruiting participants –Access –Building trust Ethical issues –Relationship between observer and observed –Recognised ethical code –Participants’ right to withdraw at any point –Informed consent - ? –Data protection and anonymity Complementary methods –Observation –Interviews

© deSouza, Dittrich, Sharp Observation

© deSouza, Dittrich, Sharp A matter of legitimacy One of the most important data collection tools of an ethnographer. Originally, it was almost a requirement for ethnographic research. It provides legitimacy: –“To say the Yanamami do this and the Borroro do that, and to be taken seriously, you have to have been there, seen them, and if not done it and brought back the T-shirt, at least captured and recorded their lives (these days on videotape). Fieldwork became de rigeur for ethnography and every practising anthropologist worth his or her salt had “his” (or “her”) tribe. This, then, is a cardinal thing to note about ethnography. Its practice is a particular form of legitimation. Ethnographers “know” in ways others don’t and can’t. And what they know derives in part from personal experience.” (Genevieve Bell, May 2004) As another data collection method it is helpful to triangulate the data.

© deSouza, Dittrich, Sharp Types According to the researcher engagement: –Participant: the researcher acts as the informants: writes code, attends meetings, discusses solutions. Example: Sharp et. al. (2004) –Non-Participant: the researcher only observes the informants. He is “fly in the wall”. Example: De Souza et. al. (2004); In any case, the researcher must document as much information as possible in his/her field notes;

© deSouza, Dittrich, Sharp Field Notes A description of events, people, interactions, tool usage, things listened, heard, experiences, impressions; Be as detailed as possible, i.e., write down the higher number of details possible; Separate observations and quotes from the informants from impressions and comments from the researcher; A private document that can only be shared within the research team;

© deSouza, Dittrich, Sharp What to collect? Or the level of detail to collect? The researcher needs to decide the level of detail that he needs his observations (Randall and Rouncefield 2004, p. 64). –Gestures, eye gazes, movements as done by (Heath and Luff, 1992) in their study of subway control rooms; –Time spent in each activity performed by knowledge workers (Gonzales and Mark, 2004) and software developers (Goncalves, de Souza et al, 2009) This level of detail depends on the research questions. Every researcher needs to define what he/she needs.

© deSouza, Dittrich, Sharp Practical Tips Write your fieldnotes, if possible, during the data collection [in the field], or at the end of the day; In the beginning of your study, write down as much as possible to the level of detail you need: –Meetings, interactions, tool usage, phone calls, process adherence, etc. Everything or anything can be relevant at this point. Often, at this point you do not know what is or is not important to your research Later, as your research progresses, field notes can be refined to focus on the “interesting” aspects only.

© deSouza, Dittrich, Sharp It is important that the ethnographer is introduced to the informants by a respected member of the group. In the beginning, there is a natural “resistance” to the ethnographer, but this will cease as time progresses; Often, ethnographers have “key informants”, i.e., experienced informants that introduce the ethnographer, facilitate his access, answer questions, provide explanations, and so on. –Finding and cultivating them is important!

© deSouza, Dittrich, Sharp Structuring frameworks to guide observation - The person. Who? - The place. Where? - The thing. What? The Goetz and LeCompte (1984) framework: - Who is present? - What is their role? - What is happening? - When does the activity occur? - Where is it happening? - Why is it happening? - How is the activity organized?

© deSouza, Dittrich, Sharp When should I stop observing? No obvious ending. Schedules often dictate when your study ends. Otherwise, stop when you stop learning new things. –when you start to see similar patterns of behavior being repeated, or –when you have listened to all the main stakeholder groups and understand their perspectives.

© deSouza, Dittrich, Sharp What data to collect? (1) The following illustrative list (Crabtree, 2003, p. 53): Activity or job descriptions. Rules and procedures (etc.) said to govern particular activities. Descriptions of activities observed. Recordings of the talk taking place between parties involved in observed activities. Informal interviews with participants explaining the detail of observed activities. Diagrams of the physical layout, including the position of artefacts.

© deSouza, Dittrich, Sharp What data to collect? (2) Photographs of artifacts (documents, diagrams, forms, computers, etc.) used in the course of observed activities. Videos of artifacts as used in the course of observed activities. Descriptions of artifacts used in the course of observed activities. Workflow diagrams showing the sequential order of tasks involved in observed activities. Process maps showing connections between activities.

© deSouza, Dittrich, Sharp Data recording Notes, audio, video, photographs Notes plus photographs Audio plus photographs Video

© deSouza, Dittrich, Sharp Comparison of the three main approaches to data recording (1) CriterionNotes plus cameraAudio plus cameraVideo EquipmentPaper, pencil, and still camera. Handheld recorder with a good microphone. Headset useful for easy transcription. Video camera – handheld or fixed. Editing, mixing, and analysis equipment needed. Flexibility of use Very flexible. Unobtrusive. Flexible. Relatively unobtrusive. Needs positioning carefully to capture relevant activity. Obtrusive. Completeness of data Only get what note taker thinks is important and can record in the time available. Visual data is missing. Notes, photographs, and sketches can augment recording but need to be coordinated with audio recording for analysis. Collects thorough and detailed data, especially if more than one camera is used, but video material needs to be coordinated for analysis.

© deSouza, Dittrich, Sharp Comparison of the three main approaches to data recording (2) CriterionNotes plus cameraAudio plus cameraVideo Disturbance to users Very low.Low, but microphone needs to be positioned. Medium. Camera can be difficult to ignore. Preliminary field studies and piloting with participants can help to lessen the impact. Reliability of data May be low. Relies on humans making a good record and knowing what to record. High but external noise, e.g. fans in computers, can muffle what is said. Captures detail of relevant activities, provided camera is positioned appropriately. AnalysisTranscription straightforward. Rich descriptions. Transcribing data can be onerous or a useful first step in data analysis. Critical discussions can be identified. Transcription needed for detailed analysis. Can revisit permanent original record. Critical incidents can be identified and tagged. Software tools for detailed analysis. Can revisit permanent original record.

© deSouza, Dittrich, Sharp Interviews

© deSouza, Dittrich, Sharp Interviews An approach for data collection used by different research methods, including ethnographic, qualitative and even quantitative; There are different types of interviews and each one of these types is more often associated with a particular research method

© deSouza, Dittrich, Sharp Types of interviews In general, interviews can be classified along two continua (DeWalt and DeWalt, 2002): –The degree of control by the researcher and informants; and –The degree to which the stimuli (questions) presented to each informant are uniform.

© deSouza, Dittrich, Sharp Degree of control by researcher –In one extreme, the researcher has no control and he is a pure observer in an conversation. The informant has control about what he or she wants to talk about; and –In the other extreme, the researcher creates a questionnaire or survey with pre-defined questions with no accommodation for the concerns or understanding of an individual respondent.

© deSouza, Dittrich, Sharp Degree to which questions to each informant are uniform –In one extreme the survey is defined beforehand and is presented to all informants with no room for his/her specific concerns; and –In the other end of the continuum, each conversation is unique, there is no attempt to ask the same questions in the same way to each informant.

© deSouza, Dittrich, Sharp Types of Questions The distinction among the types of interviews is closely linked to the type of questions that are asked: –Closed questions: well-defined questions with a limited number of answers. The informant has to choose among this set of answers; –Open questions: ill-defined questions that require an opinion, an explanation, etc.

© deSouza, Dittrich, Sharp Example of Closed Questions Please indicate your level of agreement for the following statement: “My colleagues provide timely information about their changes in the source code that affect my work”. a.Strongly disagree b.Disagree c.Neutral d.Agree e.Strongly agree

© deSouza, Dittrich, Sharp Example of Open Questions Can you explain me what do you do as a software architect? Please, could you sort for me the features in [the current prototype] that you think would make your own work more effective for the group?

© deSouza, Dittrich, Sharp Ask simple questions first Start the interview with the simple questions. In general, questions that require the informant to trust you should not be used in the beginning of the interview. Build rapport first. Even, if there already is rapport between you and him, ask simple questions first;

© deSouza, Dittrich, Sharp Ask short questions Instead of asking long and confuse questions, ask short questions: “tell me about your day, what did you do today?” “Grand tour” questions are questions that ask an informant to describe his reality, to provide an overview of his work –Can you explain me what do you do as a software architect? Based on the informant’s answer, you can follow up on different aspects of his work.

© deSouza, Dittrich, Sharp Avoid Generalizations When the informant generalize something (“I am usually late in my coding activities”), ask him to be specific (“Give me an example.” “When was the last time you were late in your code activities”). –Specific cases often provide important insights about one’s work; –You can also explore why he believes in the generalization

© deSouza, Dittrich, Sharp Naive Questions (play dumb!) Over time the researcher becomes more and more familiar with the field. Despite that, he should avoid start making assumptions about what he knows about the field, he should not stop asking simple, even naive questions; –Avoid questions that confirm what you think you know! Everyone here uses Eclipse, right? –Instead ask the informant to explain something again to you, so that you can confirm what you think you know. Which software development tools are used here?

© deSouza, Dittrich, Sharp A large part of doing good field work is that ability to make yourself vulnerable and slightly stupid. That willingness to suspend early assumptions about the way the world works, and to let other people's ways of making meaning be the ones that make sense for you. (Genevieve Bell, May 2004)

© deSouza, Dittrich, Sharp Naive Questions (3) If you have a CS degree or similar and is doing fieldwork in Software Engineering, the informants might make assumptions about what you already know (compilers, programming languages, etc). So, depending on your interest, you might even pretend to not know some of these aspects, i.e., play dumb!

© deSouza, Dittrich, Sharp Active Listening The researcher is primarily a listener who gains information in the terms used by the informants as the informant answers the questions. –This means, the researcher interrupts the informant as little as possible, and only when he thinks there is an interesting aspect that is not fully developed by the informant

© deSouza, Dittrich, Sharp Active Listening (2) The researcher is quiet, but is active. –He is aware of what is being discussed; –He is making [mental or jot] notes about what is said, who said it and whether that makes sense in the context of the project; –He is demonstrating his interest to what is being said and his respect to the ideas of the informant;

© deSouza, Dittrich, Sharp Sensitive Silence In some cultures, just 1 or 2 seconds of silence is enough to encourage informants to continue talking without the need of the researcher to intervene; –This silent can be used as a way to let an informant expand on a particular topic; Note that being quiet during an interview should not be underestimated, this in fact, is one of the more difficult skills to learn as one learns interviewing [DeWalt and DeWalt, 2002]

© deSouza, Dittrich, Sharp The Uh-hu Prompt “Uh-hu”, “hmmm-hmmm”, “ok”, “yes”, “really” are all prompts to active listening as a way of saying to the informant: –“I am following you”, or “This is interesting”, or “Please go on”

© deSouza, Dittrich, Sharp Repetition Feedback This basically is to repeat the last word or comment by the informant to stimulate him to continue the conversation and to demonstrate that the researcher is following the conversation, is interested –Be careful to not sound like a parrot ;-)

© deSouza, Dittrich, Sharp Summary Feedback According to this technique, the researcher simply repeats what has been said by the informant to confirm his understanding and to encourage the informant to continue and make corrections;

© deSouza, Dittrich, Sharp Tell me more! “Tell me more!”. Use questions and comments that encourage the informant to continue talking about what happened, when happened, who was involved, etc: –Tell me more! –What else? –What happened? –What did you do? The idea here is to encourage the informant to keep talking about the same topic, instead of encourage him to change topics;

© deSouza, Dittrich, Sharp Practical Aspects The informant’s time is precious. 1.Use wisely –Do no plan interviews longer than one hour. People will be too tired after 45 minutes; 2.Be careful: –Always have extra time for your interviews, i.e., save 45 minutes for a 30-minute interview; –Choose a quiet place without interruptions; –Ask permission to tape the interview. If that is acceptable, take extra batteries and tapes with you. Otherwise, take notes, jot down reminders without interrupting the informant.

© deSouza, Dittrich, Sharp If you cannot write notes during the interviews, do it less than 24 hours afterwards. After that, details get lost; Write a summary of the interview: –What was interesting? –What was not interesting? –What did you learn different? –What was unexpected? –What would you do differently in the next interview? These summaries will allow you to find relevant information fast and reflect about the next interview.

© deSouza, Dittrich, Sharp Quality assurance Set goals –Know what your focus is Pilot study –Learn from your mistakes Member checking –Replay your observations – you may have it wrong (e.g. red bar) Triangulation –By data collection method –By analysis approach

© deSouza, Dittrich, Sharp Practical Exercise This exercise will give attendees a chance to try out observation for themselves

© deSouza, Dittrich, Sharp Writing ethnography An ethnography is a “post hoc” representation or account of what has been seen, heard and found “in the field”. Writing the ethnography is not just “writing up” the field notes. It involves their interpretation and analysis. (Dourish 1994)

© deSouza, Dittrich, Sharp Data analysis Writing ethnography –Narrative Identifying Themes –Confirming and disconfirming –Many voices –Do I have to use grounded theory? Grounded Theory –Building a theory – what does it look like? Theoretical lenses –Distributed cognition, activity theory, …?

© deSouza, Dittrich, Sharp 45 Grounded Theory Authored by Glaser and Strauss in 1967; It does not require a prior theory about the data, that is, a set of hypotheses to be tested. –Instead, the goal of grounded theory is precisely to generate theory grounded exclusively on the existing data. It aims to develop a theory or explanation about what is going on in the field, or more specifically, what is available in the data collected. [Glaser & Strauss, 1967] and [Strauss & Corbin, 1997]

© deSouza, Dittrich, Sharp Grounded Theory: Overview Grounded theory is based on Coding, which is the analysis of the data; Field notes and transcriptions of interviews are coded to identify concepts and categories: –A concept names a phenomenon. It abstracts an event, object, or action/interaction that is significant to the researcher [Strauss and Corbin, 1998; p103]. –Categories are grouping concepts put together under a more abstract high order concept [Strauss and Corbin, 1998; p113]. 46

© deSouza, Dittrich, Sharp 47 Grounded Theory: coding Open coding –data is micro-analyzed (line-by-line) to identify categories Axial coding –categories were broken into subcategories. Whereas categories stand for phenomena, subcategories answer questions about the phenomenon, such as when, where, why, who, how, and with what consequences; –Identifies relationships between categories

© deSouza, Dittrich, Sharp 48 Grounded Theory: coding Selective coding –the most important categories are selected to be core categories, that is, the categories that will be used to describe the emerging theory

© deSouza, Dittrich, Sharp 49 Grounded Theory: Open Coding

© deSouza, Dittrich, Sharp Distributed cognition Used for analysing collaborative work –Ed Hutchins ship navigation and aeroplane flying –Ackerman and Halverson call centres –Adapted for use in HCI The people, environment & artefacts are regarded as one cognitive system Focuses on information propagation & transformation Can be used to answer ‘what if’ questions & breakdowns

© deSouza, Dittrich, Sharp Practical Exercise This exercise will ask attendees to return to the data used in the morning and consider coding from a different perspective

© deSouza, Dittrich, Sharp Group Discussion This exercise will be a chance to consider what is/may be different about applying ethnography in software engineering contexts