1. Chapter 7
Data gathering

2. Outline Four key issues Data recording Interviews Questionnaires
Observation
Choosing and combining techniques

3. Four key issues Goal setting
Relationship between data collector and data provider
Triangulation
Pilot studies

4. Setting goals Examples:
To understand how technology fits into normal family life
To identify which of two icons representing a ‘send ’ action is easier to use
To find out whether the redesign you are planning for a hand-held meter reader is along the right lines

5. Setting goals Identify specific goals for the particular study
Goals will influence the nature of the data gathering sessions, the techniques, and also the analysis
Goals should be clear and concise

6. Relationship with participants
Making sure that this relationship is clear and professional
Ask participants to sign an informed consent form
Incentives for participating in the data gathering sessions might be needed in circumstances that there is no clear and direct advantage to the participants

7. Triangulation
Using more than one data gathering / data analysis technique
Examples:
Using observation to understand the context of task performance
Interviews to target specific user groups
Questionnaires to reach a wider population
Provides different perspectives and corroboration of findings across techniques

8. Pilot studies A small trial run of the main study
Plans for a data gathering session should be tested by doing a pilot study before launching into the main study
Test the equipment and instructions
Questions can be tested for clarity
Experimental procedure can be confirmed as viable
Anyone involved in a pilot study cannot be involved in the main study

9. Data recording
Which techniques are used will depend on the context, time available, and the sensitivity of the situation
The choice of data recording techniques will have an impact on how intrusive the data gathering will be

10. Data recording Three common techniques are Notes plus still camera
Audio plus still camera
Audio is less intrusive than video
Video
Capture both visual and audio data but can be intrusive

11. Interviews Types of interviews Open-ended or unstructured Structured
Semi-structured
Group interviews

12. Unstructured interviews
Exploratory, to explore the range of opinions
No particular expectation about the format or content of answers
Example: What are the advantages of using a PDA?
Both interviewer and interviewee can steer the interview
A balance between relevant answers and following unanticipated new lines of enquiry

13. Unstructured interviews
Generate rich data
Interviewees may mention issues that the interviewer has not considered
Data analysis is time-consuming
Impossible to replicate the process

14. Structured interviews
Interviewer asks predetermined questions
Useful when the goals are clearly understood and specific questions can be identified
Questions are closed, meaning they require an answer from a predetermined set of alternatives
For each participant, question wording should be exactly the same and questions are asked in the same order

15. Structured interviews
Examples:
Which of the following websites do you visit most frequently: amazon.com, google.com, msn.com?
How often do you visit this website: everyday, once a week, once a month, less often than once a month?
Have you ever purchased anything online?
If so, how often do you purchase items online: every day, once a week, once a month, less often than once a month?

16. Semi-structured interviews
Use both closed and open questions
For consistency, a basic script is used for guidance so that the same topics are covered with each interviewee
The interviewer starts with preplanned questions and then probes the interviewee to say more until no new relevant information is forthcoming

17. Semi-structured interviews
Example:
Which music websites do you visit most frequently? (hottestmusic.com)
Why? (site layout)
Tell me more about the site layout
Anything else that you like about the site?
Thanks. Are there any other reasons for visiting this site so often that you haven’t mentioned?

18. Focus groups Interview people in groups
Normally, 3 to 10 people are involved, and the discussion is led by a trained facilitator
In requirements activities, to identify conflicts in terminology or expectations from different sections within one organization
The method assumes that individuals develop opinions within a social context by talking with others

19. Planning and conducting an interview
Developing questions or topics to be covered
Collating any documentation to give to the interviewee
Checking that recording equipment works in advance and you know how to use it
Working out the structure of the interview
Organizing a suitable time and place

20. Developing interview questions
Two types:
‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’
‘open questions’ do not have a predetermined format
Closed questions are easier to analyze
From:

21. Developing interview questions
Split compound sentences into separate questions
Avoid using technical terms or complex language
Try to keep questions neutral
Example: “Why do you like this style of interaction? -> assumes the person does like it and will discourage some interviewees from stating their real feelings

22. Running the interview Introduction interviewer introduces himself
explains why he is doing the interview
reassures interviewees regarding any ethical issues
asks if they mind being recorded, if appropriate

23. Running the interview
Warm-up session starts with easy, non-threatening questions, such as questions about demographic information
Main session in which the questions are presented in a logical sequence, with the more probing ones at the end

24. Running the interview
Cool-off period consists of a few easy questions to ease tension if it has arisen
Closing session in which the interviewer thanks the interviewee and switches off the recorder or puts her notebook away

25. Other forms of interview
Telephone interviews
Online interviews
asynchronous communication, e.g.
synchronous communication, e.g. IM

26. Enriching the interview experience
Interviews often take place in a neutral environment, e.g. a meeting room away from the interviewee’s normal desk
Interviews can be enriched by using props such as prototypes or work artifacts that the interviewee or interviewer brings along

27. Questionnaires Questions can be closed or open
Closed questions are easier to analyze, and may be done by computer
Can be administered to large populations
Paper, and the web used for dissemination
Sampling can be a problem when the size of a population is unknown as is common online
From:

28. Questionnaires Start by asking for basic demographic information
Question order
Different versions of the questionnaire for different populations
Clear instructions on how to complete the questionnaire

29. Questionnaires
A balance between using white space and the need to keep the questionnaire as compact as possible
Decide on whether phrases will all be positive, all negative or mixed.

30. Question and response format
Check boxes and ranges
Rating scales
Likert scales
Semantic differential scales
Open-ended responses

31. Likert scales
1. Gather a pool of short statements about the subject to be investigated. For example, “This control panel is easy to use”
2. Decide on the scale
How many points does the scale need?
Should the scale be discrete or continuous?
How to represent the scale?
3. Select items for the final questionnaire and reword as necessary to make them clear

32. Semantic differential scales
Used less frequently than Likert scales
Explore a range of bipolar attitudes about a particular item
Each pair of attitudes is represented as a pair of adjectives
For example:
Attractive _ _ _ _ _ _ _ Ugly
Clear _ _ _ _ _ _ _ Confusing

33. What scales to use – 3, 5, 7 or more?
Use a small number, e.g. 3, when possibilities are very limited, as in yes/no type answers
Use a medium-sized range, e.g. 5, when making judgments that involve like/dislike, agree/disagree statements
Use a longer range, e.g. 7 or 9, when asking respondents to make subtle judgments. For example, when asking about a ‘level of appeal’ of a character in a video game

34. What order?
Place the positive end of the scale first and the negative end last. This matches the logical way people think about scoring. For example:
__ strongly agree
__ slightly agree
__ agree
__ slightly disagree
__ strongly disagree

35. Encouraging a good response
Make sure purpose of study is clear
Promise anonymity
Ensure questionnaire is well designed
Offer a short version for those who do not have time to complete a long questionnaire
If mailed, include a stamped addressed envelope
Follow-up with s, phone calls, letters
Provide an incentive
40% response rate is high, 20% is often acceptable
From:

36. Online questionnaires
1. Devise the questionnaire as if it is to be delivered on
paper first
2. Develop strategies for reaching the target
population
3. Produce an error-free interactive electronic version
4. Make the questionnaire accessible from all common
browsers and readable from different sized monitors
and different network locations

37. Online questionnaires
5. Make sure information identifying each respondent can be captured and stored confidentially because the same person may submit several completed surveys
6. Thoroughly pilot test the questionnaire

38. Advantages of online questionnaires
Responses are usually received quickly
No copying and postage costs
Data can be collected in database for analysis
Time required for data analysis is reduced
Errors can be corrected easily
From:

39. Problems with online questionnaires
Sampling is problematic if population size is unknown
Preventing individuals from responding more than once
Individuals have also been known to change questions in questionnaires
From:

40. Observation Direct observation in the field
Direct observation in controlled environments
Indirect observation: tracking users’ activities

41. Direct observation in the field
There is a careful balance between being guided by goals and being open to modifying, shaping, or refocusing the study as you learn about the situation
Being able to keep this balance is a skill that develops with experience

42. Structuring frameworks for observation in the field
Easy framework
The person – who is using the technology at any particular time?
The place – where are they using it?
The thing – what are they doing with it?

43. Structuring frameworks for observation in the field
More detailed framework – pay greater attention to the context of the activity (Colin Robson, 2002)
Space – what is the physical space like and how is it laid out?
Actors – what are the names and relevant details of the people involved?
Activities – what are the actors doing and why?

44. Structuring frameworks for observation in the field
Objects – what physical objects are present, such as furniture?
Acts – what are specific individual actions?
Events – is what you observe part of a special event?
Time – what is the sequence of events?
Goals – what are the actors trying to accomplish?
Feelings – what is the mood of the group and of individuals?

45. Structuring frameworks for observation in the field
Specific framework to support field studies in conjunction with designing future technologies (Rogers and Bellotti, 1997)
Two parts:
Problematizing existing settings
Envisioning future settings

46. Structuring frameworks for observation in the field
Problematizing existing settings
Why is an observation about a work practice or other activity striking?
What are the pros and cons of the existing ways technologies are used in a setting?
How have ‘workarounds’ evolved and how effective are they?
Why do certain old-fashioned practices, using seemingly antiquated technologies persist, despite more advanced technologies being available in the setting?

47. Structuring frameworks for observation in the field
Envisioning future settings
What would be gained and lost through changing current ways of working or carrying out an activity by introducing new kinds of technological support?
What might be the knock-on effects for other practices and activities through introducing new technologies?
How might other settings be enhanced and disrupted through deploying the same kinds of future technologies?

48. Degree of participation
Insider = Participant observer
Attempt to become a full member of the group he is studying
Important to keep the two roles clear and separate
Observation notes are objective, while participation is also maintained
Outsider = Passive observer
Not take any part in the study environment at all

49. Planning and conducting an observation in the field
Choosing a framework
Level of participation
How to record the data
How to gain acceptance in the group being studied

50. Planning and conducting an observation in the field
How to handle sensitive issues such as cultural differences
How to ensure that the study uses different perspectives (people, activities, job roles, etc.)
Work as a team
Each person can focus on different people or parts
Observation and reflection can be easily interweaved
Observations can be compared -> more reliable data
Results will reflect different perspectives

51. Planning and conducting an observation in the field
Everyone in the group needs to be attended to, not just those who receive you well
Need to refocus the study as you reflect upon what has been seen
Ideas will sharpen into questions that guide further observation

52. Planning and conducting an observation in the field
At the end of each day
You must check the notes and other records
You must write up experiences and observations
Writing a diary or private blog is one way of keeping up
Annotate any collected or copied documents, describing how they are used and at what stage of the activity

53. Planning and conducting an observation in the field
As notes are reviewed, personal opinion should be separated from observation of what happened
Anything for further investigation should be clearly marked
Check observations with an informant or members of the group to ensure that you have understood what is happening and that your interpretations are accurate

54. Ethnography
A large part of most ethnographic studies is direct observation, but interviews, questionnaires, and studying artifacts used in the activities also feature in many studies
It aims to observe a situation without imposing any a priori structure or framework upon it, and to view everything as ‘strange’

55. Ethnography
The observer adopts a participant observer (i.e. insider) role as much as possible
Ethnographic data: the observer gathers what is available, what is ‘ordinary,’ what it is that people do, say, how they work

56. Ethnography
Initially, time should be spent getting to know the people in the work place and bonding with them
Explain to them why you are there, what you hope to achieve, and how long you plan to be there

57. Ethnography
The observer needs to be prepared to switch in and out of people’s work cycles, moving into the shadow if something happens that needs the worker’s immediate attention
A good tactic is to explain to one of the participants what you think is happening and then let him/her correct you

58. Ethnography It is important not to appear overly keen or obtrusive
Asking too many questions, taking picture of everything, showing off your knowledge, and getting in people’s way can be very off-putting

59. Ethnography
Listening and watching while sitting on the sidelines and occasionally asking questions is a much better approach
When you have gained the trust and respect of the participants you can then ask if they mind you setting up a video camera, taking pictures, or using a recorder

60. Ethnography List of materials that might be recorded or collected
Activity or job descriptions
Rules and procedures said to govern particular activities
Descriptions of activities observed
Recordings of the talk taking place between parties involved in observed activities

61. Ethnography
Informal interviews with participants explaining the detail of observed activities
Diagrams of the physical layout, including the position of artifacts
Photographs of artifacts (documents, diagrams, forms, computers, etc.) used in the course of observed activities

62. Ethnography
Videos of artifacts as used in the course of observed activities
Description of artifacts used in the course of observed activities
Workflow diagrams showing the sequential order of tasks involved in observed activities
Process maps showing connections between activities

63. Applied ethnographic research
Ann Rose et al. developed a procedure to be used by technical design teams with limited ethnographic training
This applied form of ethnography acknowledges the comparatively small amounts of time available for any kind of user study

64. Applied ethnographic research
4 stages:
1) Preparation
Understand organization policies and work culture
Familiarize yourself with the system and its history
Set initial goals and prepare questions
Gain access and permission to observe and interview

65. Applied ethnographic research
2) Field study
Establish a rapport with managers and users
Observe and interview users in their workplace and collect data
Follow any leads that emerge from the visits
Record your visits

66. Applied ethnographic research
3) Analysis
Compile the collected data in numerical, textual, and multimedia databases
Quantify data and compile statistics
Reduce and interpret the data
Refine the goals and processes used

67. Applied ethnographic research
4) Reporting
Consider multiple audiences and goals
Prepare a report and present the findings

68. Direct observation in controlled environment
Prepare a script to guide how the participants will be greeted, be told about the goals of the study, and how long it will last, and have their rights explained
In the laboratory, the emphasis is on the details of what individuals do
In the field, the context is important and the focus is on how people interact with each other, the technology, and their environment

69. Direct observation in controlled environment
Think-aloud technique
A useful way of understanding what is going on in a person’s head
Requires people to say out loud everything that they are thinking and trying to do, so that their thought processes are externalized

70. Indirect observation: tracking users’ activities
Techniques:
Diaries
Interaction logs

71. Indirect observation: tracking users’ activities
Diaries
Participants are asked to write a diary of their activities on a regular basis, e.g. what they did, when they did it, what they found hard or easy, and what their reactions were to the situation

72. Indirect observation: tracking users’ activities
Advantages of diaries
Diaries are useful when participants are scattered and unreachable in person
Not take up much resource
No special equipment or expertise
Suitable for long-term studies
Online data collection, enabling the data to go straight into a database

73. Indirect observation: tracking users’ activities
Disadvantages of diaries
Rely on participants being reliable and remembering to complete diaries
Participants’ memories of events are often exaggerated
Incentives may be needed
The process has to be straightforward and quick

74. Indirect observation: tracking users’ activities
Interaction logs
Involves instrumenting the software to record users’ activity in a log that can be examined later

75. Indirect observation: tracking users’ activities
Advantages
it is unobtrusive, but it also raises ethical concerns about observing participants without their knowledge
Large volume of data can be logged automatically. However, powerful tools are needed to explore and analyze this data quantitatively and qualitatively
Spyware

76. Choosing and combining techniques
Depends on
The focus of the study
The participants involved
The nature of the technique
The resources available

77. Summary
Three main data gathering methods: interviews, questionnaires, observation
Four key issues of data gathering: goals, triangulation, participant relationship, pilot
Interviews may be structured, semi-structured or unstructured
Questionnaires may be on paper, online or telephone
Observation may be direct or indirect, in the field or in controlled setting
Techniques can be combined depending on study focus, participants, nature of technique and available resources
From: