Chapter 7 Data gathering

Outline Four key issues Data recording Interviews Questionnaires Observation Choosing and combining techniques

Four key issues Goal setting Relationship between data collector and data provider Triangulation Pilot studies

Setting goals Examples: To understand how technology fits into normal family life To identify which of two icons representing a ‘send email’ action is easier to use To find out whether the redesign you are planning for a hand-held meter reader is along the right lines

Setting goals Identify specific goals for the particular study Goals will influence the nature of the data gathering sessions, the techniques, and also the analysis Goals should be clear and concise

Relationship with participants Making sure that this relationship is clear and professional Ask participants to sign an informed consent form Incentives for participating in the data gathering sessions might be needed in circumstances that there is no clear and direct advantage to the participants

Triangulation Using more than one data gathering / data analysis technique Examples: Using observation to understand the context of task performance Interviews to target specific user groups Questionnaires to reach a wider population Provides different perspectives and corroboration of findings across techniques

Pilot studies A small trial run of the main study Plans for a data gathering session should be tested by doing a pilot study before launching into the main study Test the equipment and instructions Questions can be tested for clarity Experimental procedure can be confirmed as viable Anyone involved in a pilot study cannot be involved in the main study

Data recording Which techniques are used will depend on the context, time available, and the sensitivity of the situation The choice of data recording techniques will have an impact on how intrusive the data gathering will be

Data recording Three common techniques are Notes plus still camera Audio plus still camera Audio is less intrusive than video Video Capture both visual and audio data but can be intrusive

Interviews Types of interviews Open-ended or unstructured Structured Semi-structured Group interviews

Unstructured interviews Exploratory, to explore the range of opinions No particular expectation about the format or content of answers Example: What are the advantages of using a PDA? Both interviewer and interviewee can steer the interview A balance between relevant answers and following unanticipated new lines of enquiry

Unstructured interviews Generate rich data Interviewees may mention issues that the interviewer has not considered Data analysis is time-consuming Impossible to replicate the process

Structured interviews Interviewer asks predetermined questions Useful when the goals are clearly understood and specific questions can be identified Questions are closed, meaning they require an answer from a predetermined set of alternatives For each participant, question wording should be exactly the same and questions are asked in the same order

Structured interviews Examples: Which of the following websites do you visit most frequently: amazon.com, google.com, msn.com? How often do you visit this website: everyday, once a week, once a month, less often than once a month? Have you ever purchased anything online? If so, how often do you purchase items online: every day, once a week, once a month, less often than once a month?

Semi-structured interviews Use both closed and open questions For consistency, a basic script is used for guidance so that the same topics are covered with each interviewee The interviewer starts with preplanned questions and then probes the interviewee to say more until no new relevant information is forthcoming

Semi-structured interviews Example: Which music websites do you visit most frequently? (hottestmusic.com) Why? (site layout) Tell me more about the site layout Anything else that you like about the site? Thanks. Are there any other reasons for visiting this site so often that you haven’t mentioned?

Focus groups Interview people in groups Normally, 3 to 10 people are involved, and the discussion is led by a trained facilitator In requirements activities, to identify conflicts in terminology or expectations from different sections within one organization The method assumes that individuals develop opinions within a social context by talking with others

Planning and conducting an interview Developing questions or topics to be covered Collating any documentation to give to the interviewee Checking that recording equipment works in advance and you know how to use it Working out the structure of the interview Organizing a suitable time and place

Developing interview questions Two types: ‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’ ‘open questions’ do not have a predetermined format Closed questions are easier to analyze From: www.id-book.com

Developing interview questions Split compound sentences into separate questions Avoid using technical terms or complex language Try to keep questions neutral Example: “Why do you like this style of interaction? -> assumes the person does like it and will discourage some interviewees from stating their real feelings

Running the interview Introduction interviewer introduces himself explains why he is doing the interview reassures interviewees regarding any ethical issues asks if they mind being recorded, if appropriate

Running the interview Warm-up session starts with easy, non-threatening questions, such as questions about demographic information Main session in which the questions are presented in a logical sequence, with the more probing ones at the end

Running the interview Cool-off period consists of a few easy questions to ease tension if it has arisen Closing session in which the interviewer thanks the interviewee and switches off the recorder or puts her notebook away

Other forms of interview Telephone interviews Online interviews asynchronous communication, e.g. email synchronous communication, e.g. IM

Enriching the interview experience Interviews often take place in a neutral environment, e.g. a meeting room away from the interviewee’s normal desk Interviews can be enriched by using props such as prototypes or work artifacts that the interviewee or interviewer brings along

Questionnaires Questions can be closed or open Closed questions are easier to analyze, and may be done by computer Can be administered to large populations Paper, email and the web used for dissemination Sampling can be a problem when the size of a population is unknown as is common online From: www.id-book.com

Questionnaires Start by asking for basic demographic information Question order Different versions of the questionnaire for different populations Clear instructions on how to complete the questionnaire

Questionnaires A balance between using white space and the need to keep the questionnaire as compact as possible Decide on whether phrases will all be positive, all negative or mixed.

Question and response format Check boxes and ranges Rating scales Likert scales Semantic differential scales Open-ended responses

Likert scales 1. Gather a pool of short statements about the subject to be investigated. For example, “This control panel is easy to use” 2. Decide on the scale How many points does the scale need? Should the scale be discrete or continuous? How to represent the scale? 3. Select items for the final questionnaire and reword as necessary to make them clear

Semantic differential scales Used less frequently than Likert scales Explore a range of bipolar attitudes about a particular item Each pair of attitudes is represented as a pair of adjectives For example: Attractive _ _ _ _ _ _ _ Ugly Clear _ _ _ _ _ _ _ Confusing

What scales to use – 3, 5, 7 or more? Use a small number, e.g. 3, when possibilities are very limited, as in yes/no type answers Use a medium-sized range, e.g. 5, when making judgments that involve like/dislike, agree/disagree statements Use a longer range, e.g. 7 or 9, when asking respondents to make subtle judgments. For example, when asking about a ‘level of appeal’ of a character in a video game

What order? Place the positive end of the scale first and the negative end last. This matches the logical way people think about scoring. For example: __ strongly agree __ slightly agree __ agree __ slightly disagree __ strongly disagree

Encouraging a good response Make sure purpose of study is clear Promise anonymity Ensure questionnaire is well designed Offer a short version for those who do not have time to complete a long questionnaire If mailed, include a stamped addressed envelope Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often acceptable From: www.id-book.com

Online questionnaires 1. Devise the questionnaire as if it is to be delivered on paper first 2. Develop strategies for reaching the target population 3. Produce an error-free interactive electronic version 4. Make the questionnaire accessible from all common browsers and readable from different sized monitors and different network locations

Online questionnaires 5. Make sure information identifying each respondent can be captured and stored confidentially because the same person may submit several completed surveys 6. Thoroughly pilot test the questionnaire

Advantages of online questionnaires Responses are usually received quickly No copying and postage costs Data can be collected in database for analysis Time required for data analysis is reduced Errors can be corrected easily From: www.id-book.com

Problems with online questionnaires Sampling is problematic if population size is unknown Preventing individuals from responding more than once Individuals have also been known to change questions in email questionnaires From: www.id-book.com

Observation Direct observation in the field Direct observation in controlled environments Indirect observation: tracking users’ activities

Direct observation in the field There is a careful balance between being guided by goals and being open to modifying, shaping, or refocusing the study as you learn about the situation Being able to keep this balance is a skill that develops with experience

Structuring frameworks for observation in the field Easy framework The person – who is using the technology at any particular time? The place – where are they using it? The thing – what are they doing with it?

Structuring frameworks for observation in the field More detailed framework – pay greater attention to the context of the activity (Colin Robson, 2002) Space – what is the physical space like and how is it laid out? Actors – what are the names and relevant details of the people involved? Activities – what are the actors doing and why?

Structuring frameworks for observation in the field Objects – what physical objects are present, such as furniture? Acts – what are specific individual actions? Events – is what you observe part of a special event? Time – what is the sequence of events? Goals – what are the actors trying to accomplish? Feelings – what is the mood of the group and of individuals?

Structuring frameworks for observation in the field Specific framework to support field studies in conjunction with designing future technologies (Rogers and Bellotti, 1997) Two parts: Problematizing existing settings Envisioning future settings

Structuring frameworks for observation in the field Problematizing existing settings Why is an observation about a work practice or other activity striking? What are the pros and cons of the existing ways technologies are used in a setting? How have ‘workarounds’ evolved and how effective are they? Why do certain old-fashioned practices, using seemingly antiquated technologies persist, despite more advanced technologies being available in the setting?

Structuring frameworks for observation in the field Envisioning future settings What would be gained and lost through changing current ways of working or carrying out an activity by introducing new kinds of technological support? What might be the knock-on effects for other practices and activities through introducing new technologies? How might other settings be enhanced and disrupted through deploying the same kinds of future technologies?

Degree of participation Insider = Participant observer Attempt to become a full member of the group he is studying Important to keep the two roles clear and separate Observation notes are objective, while participation is also maintained Outsider = Passive observer Not take any part in the study environment at all

Planning and conducting an observation in the field Choosing a framework Level of participation How to record the data How to gain acceptance in the group being studied

Planning and conducting an observation in the field How to handle sensitive issues such as cultural differences How to ensure that the study uses different perspectives (people, activities, job roles, etc.) Work as a team Each person can focus on different people or parts Observation and reflection can be easily interweaved Observations can be compared -> more reliable data Results will reflect different perspectives

Planning and conducting an observation in the field Everyone in the group needs to be attended to, not just those who receive you well Need to refocus the study as you reflect upon what has been seen Ideas will sharpen into questions that guide further observation

Planning and conducting an observation in the field At the end of each day You must check the notes and other records You must write up experiences and observations Writing a diary or private blog is one way of keeping up Annotate any collected or copied documents, describing how they are used and at what stage of the activity

Planning and conducting an observation in the field As notes are reviewed, personal opinion should be separated from observation of what happened Anything for further investigation should be clearly marked Check observations with an informant or members of the group to ensure that you have understood what is happening and that your interpretations are accurate

Ethnography A large part of most ethnographic studies is direct observation, but interviews, questionnaires, and studying artifacts used in the activities also feature in many studies It aims to observe a situation without imposing any a priori structure or framework upon it, and to view everything as ‘strange’

Ethnography The observer adopts a participant observer (i.e. insider) role as much as possible Ethnographic data: the observer gathers what is available, what is ‘ordinary,’ what it is that people do, say, how they work

Ethnography Initially, time should be spent getting to know the people in the work place and bonding with them Explain to them why you are there, what you hope to achieve, and how long you plan to be there

Ethnography The observer needs to be prepared to switch in and out of people’s work cycles, moving into the shadow if something happens that needs the worker’s immediate attention A good tactic is to explain to one of the participants what you think is happening and then let him/her correct you

Ethnography It is important not to appear overly keen or obtrusive Asking too many questions, taking picture of everything, showing off your knowledge, and getting in people’s way can be very off-putting

Ethnography Listening and watching while sitting on the sidelines and occasionally asking questions is a much better approach When you have gained the trust and respect of the participants you can then ask if they mind you setting up a video camera, taking pictures, or using a recorder

Ethnography List of materials that might be recorded or collected Activity or job descriptions Rules and procedures said to govern particular activities Descriptions of activities observed Recordings of the talk taking place between parties involved in observed activities

Ethnography Informal interviews with participants explaining the detail of observed activities Diagrams of the physical layout, including the position of artifacts Photographs of artifacts (documents, diagrams, forms, computers, etc.) used in the course of observed activities

Ethnography Videos of artifacts as used in the course of observed activities Description of artifacts used in the course of observed activities Workflow diagrams showing the sequential order of tasks involved in observed activities Process maps showing connections between activities

Applied ethnographic research Ann Rose et al. developed a procedure to be used by technical design teams with limited ethnographic training This applied form of ethnography acknowledges the comparatively small amounts of time available for any kind of user study

Applied ethnographic research 4 stages: 1) Preparation Understand organization policies and work culture Familiarize yourself with the system and its history Set initial goals and prepare questions Gain access and permission to observe and interview

Applied ethnographic research 2) Field study Establish a rapport with managers and users Observe and interview users in their workplace and collect data Follow any leads that emerge from the visits Record your visits

Applied ethnographic research 3) Analysis Compile the collected data in numerical, textual, and multimedia databases Quantify data and compile statistics Reduce and interpret the data Refine the goals and processes used

Applied ethnographic research 4) Reporting Consider multiple audiences and goals Prepare a report and present the findings

Direct observation in controlled environment Prepare a script to guide how the participants will be greeted, be told about the goals of the study, and how long it will last, and have their rights explained In the laboratory, the emphasis is on the details of what individuals do In the field, the context is important and the focus is on how people interact with each other, the technology, and their environment

Direct observation in controlled environment Think-aloud technique A useful way of understanding what is going on in a person’s head Requires people to say out loud everything that they are thinking and trying to do, so that their thought processes are externalized

Indirect observation: tracking users’ activities Techniques: Diaries Interaction logs

Indirect observation: tracking users’ activities Diaries Participants are asked to write a diary of their activities on a regular basis, e.g. what they did, when they did it, what they found hard or easy, and what their reactions were to the situation

Indirect observation: tracking users’ activities Advantages of diaries Diaries are useful when participants are scattered and unreachable in person Not take up much resource No special equipment or expertise Suitable for long-term studies Online data collection, enabling the data to go straight into a database

Indirect observation: tracking users’ activities Disadvantages of diaries Rely on participants being reliable and remembering to complete diaries Participants’ memories of events are often exaggerated Incentives may be needed The process has to be straightforward and quick

Indirect observation: tracking users’ activities Interaction logs Involves instrumenting the software to record users’ activity in a log that can be examined later

Indirect observation: tracking users’ activities Advantages it is unobtrusive, but it also raises ethical concerns about observing participants without their knowledge Large volume of data can be logged automatically. However, powerful tools are needed to explore and analyze this data quantitatively and qualitatively Spyware

Choosing and combining techniques Depends on The focus of the study The participants involved The nature of the technique The resources available

Summary Three main data gathering methods: interviews, questionnaires, observation Four key issues of data gathering: goals, triangulation, participant relationship, pilot Interviews may be structured, semi-structured or unstructured Questionnaires may be on paper, online or telephone Observation may be direct or indirect, in the field or in controlled setting Techniques can be combined depending on study focus, participants, nature of technique and available resources From: www.id-book.com