Data gathering

Four key issues Setting goals Decide how to analyze data once collected Relationship with participants Clear and professional Informed consent when appropriate Triangulation Use more than one approach Pilot studies Small trial of main study The main reason for gathering data is to obtain information about something. It could be how do you like this application to whether the redesign of the website achieved its aims. Whatever the reason may be, it must be very clearly identified as these influence the nature of the data gathering session and the data gathering techniques that will be used and how we analyze the data. The goal can be expressed formally using some mathematical notation or using a simple description. It must be clear and concise. A great way to establish a good relationship is to use an informed consent form. The consent form ensures the rights of both the data gatherer(they can use the data in their analysis, publish it ) and the data provider(the data provided will not be detrimental to them). How do you work with young children? This kind of consent form is not required when we are gathering requirements. Why? Incentives will help in gathering data. Triangulation is a strategy than demands the use of more than one data gathering technique to tackle a goal or using more than one data analysis approach. Questionnaires can be sent to a larger user group, interview a specific user group and observe some sample people from that target user group. It provides different perspectives and some findings may be corroborated, leading to a better conclusion. The aim of a pilot study is to ensure that the study is viable. Potential problems can be identified in advance and corrected. It avoids wasting time, money and precious resources. Anyone involved in a pilot study cannot be used in the main one. Why? This helps avoid distorting the final study.

Data recording Notes, audio, video, photographs Notes plus photographs Audio plus photographs Video The data recording technique that you choose, depends on the context, the time available and the sensitivity of the situation. It is not good to conduct interview and take notes at the same time. Avoids personal bias(you capture only what is deemed important by the note taker). Audio is less intrusive than video, but transcribing it can be a pain. Video has the advantage of capturing both visual and audio data. The problem with video is you focus on what can be captured using the lens, that you might miss out on things happening beyond the camera view. Video is the most complete form of data gathering and is highly reliable.

Interviews Unstructured - are not directed by a script. Rich but not replicable. Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness. Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability. Interviews can be thought of as a conversation with a purpose. The above three types of interviews are different based on the amount of control that the interviews wields. Unstructured: They are real useful when you are exploring a new concept. It is open ended, but it is better to have a plan for the type of questions that you want covered. A lot of unnecessary data may be gathered, which may be very difficult to analyze. Mostly open questions Structured interviews are very useful when goals are clearly understood and specific questions can be identified. Questions in a structured interview must be worded in the exact manner for all participants. Mostly closed questions Interviews, we must make sure that we do not lead the users into a specific answer with the way we phrase our questions, body language, etc. Make sure you give enough time for people to answer. Avoid gender stereotypes and long questions, jargons etc. Choose your questions carefully to avoid bias. This is very valid, because people might sometimes give answers that show them in the best light rather than what they actually think. Follow the usability study process, communicate the goals, reassure the participants, have a exploratory period, the main session, and cool off towards the end.

Focus Groups 3-10 People People develop opinions within a social context It is commonly used in political campaigns(Spinning Boris)marketing and social research. The discussion is led by a trained facilitator and the 3-10 people invited are usually representative of the target population. It allows diverse and sensitive opinions to be revealed and people often develop opinions when they hear other people talk about a specific issue. A preset agenda with sufficient flexibility to follow unanticipated issues is needed.

Questionnaires Questions can be closed or open Closed questions are easier to analyze, and may be done by computer Paper, email and the web used for dissemination Sampling can be a problem when the size of a population is unknown as is common online Questionnaires are really useful at getting answers to specific questions from a large group of people who are in various areas. They are used on their own or can be used in conjunction with other data gathering techniques to deepen your understanding of a particular situation. How do you decide between a structured interview and a questionnaire? It all comes down to the motivation of the respondent. If the respondent is motivated to answer their questions, then you may want to use a questionnaire, but if the level of motivation is not high enough, then you might want to use a structured interview.

Questionnaire design Question Order. Clear Instructions Length Decide on whether phrases will all be positive, all negative or mixed. Do you need different versions of the questionnaire for different populations? Encourage a good response Many questionnaires typically begin by asking for basic demographic information, sort of like a pre-test questionnaire. This information is really valuable when it comes down to analyzing the results. The background information provides valuable context to the questionnaire. If two answers conflict, it could be explained by their perspectives on the situation. Make sure you collect information that is useful to the study. The impact of the questions can be influenced by the question order. Strike a balance between using white space and keeping the questionnaire compact. Make sure purpose of study is clear Promise anonymity Ensure questionnaire is well designed Offer a short version for those who do not have time to complete a long questionnaire If mailed, include a stamped addressed envelope Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often acceptable

Question and response format ‘Yes’ and ‘No’ Checkboxes that offer many options Rating scales Likert scales semantic scales 3, 5, 7 or more points? Open-ended responses Semantic differential scales are used less frequently than Likert scales. They explore bipolar attitudes to a particular item and provide a pair of words that provide different meanings. For example clear …………………………………confusing How many scale points do we use, 3 or 5 or 7. Long scales help provide a nice discrimination, while it might be difficult for people to find the difference between some scale points. An odd number provides a clear central point, while an even number forces a decision and prevents them from sitting on a fence.

Advantages of online questionnaires Responses are usually received quickly No copying and postage costs Data can be collected in database for analysis Time required for data analysis is reduced Errors can be corrected easily Email versus web-based. An email one can be targeted at very specific people, but it is just an electronic version of a paper based survey and it is not very interactive and does not provide data validation and analsyis.

Problems with online questionnaires Preventing individuals from responding more than once Individuals have also been known to change questions in email questionnaires The sampling technique with online questionnaires is not very scientific. The term to use is convenience sampling. It just involves people who were available rather than ones who fit the representative population. Some studies have shown that people are more revealing and honest when answer online questionnaires than when answering paper based questionnaires. For instance students tend to rate their profs less favorable in web based evaluation forms than on paper based ones. Why?

Observation Direct observation in the field Structuring frameworks Degree of participation (insider or outsider) Ethnography Direct observation in controlled environments Indirect observation: tracking users’ activities Diaries Interaction logging Insider vs Outsider Observation is a useful data gathering tool that can be used at any stage in the product development. It can be used early on to observer people using the current system. It gives us a good idea of the context of tool usage, the goals and the problems or it can be used at a later stage to watch people interact with a prototype. The investigator may directly observe the user or might get access to records the user makes while using the tool. Field observation can be better than observing in a usability lab, because it is possible to see the real feelings(frustration or delight) with the product. Also the user has more control in a natural setting. But because an insider is present, people may not be willing to share their true annoyances. Observation is one technique, where it is real important to have a clear goal. There are a lot of things happening and we need to be observe the right thing, but at the same time we should be flexible. An observer might choose to just watch without participating in the activity, or can choose to be a full time participant. In the latter case, you have to make sure you are objective with your results.-

Structuring frameworks to guide observation - The person. Who? - The place. Where? - The thing. What? The Goetz and LeCompte (1984) framework: - Who is present? - What is their role? - What is happening? - When does the activity occur? - Where is it happening? - Why is it happening? - How is the activity organized? Who is using it? What are they doing with it and where are they using it?

Direct observation in a controlled setting Think-aloud technique Indirect observation Diaries Interaction logs Ethical concerns with logging. When do you stop observation?

Choosing and combining techniques Depends on The focus of the study The participants involved The nature of the technique The resources available The techniques must be compatible with the focus of the study, for example the type of data you want to gather, different data gathering techniques can give you different kinds of information, do you want opinions , observable behavior or facts etc. The participants involved, children versus adults, location of the target population, time needed to complete the study, if you need a lot of time, then you might want to observe, the level of motivation can decide between questionnaires and interviews.