Gathering User Data IS 588 Dr. Dania Bilal Spring 2008

Getting to know users Setting goals for data collection Working with research participants –Consent forms (IRBs) (signed by adults) –Assent forms (signed by children) Choosing from various methods –Quantitative and qualitative

Methods Interviews Focus groups Observation Surveys and survey questionnaires Field studies (in Evaluation) Transaction logs Experiments (in Evaluation)

Interviews Structured interviews –Predetermined closed questions established Unstructured interviews –Open questions Semi-structured interviews –Open and closed questions

Planning an interview Develop a set of questions based on goals and objectives of data gathering Open vs. closed questions Combination of open and closed questions Questions to be written according to user’s level of understanding –Explanations provided as needed

Types of interviews Face-to-face individual interviews with answers handwritten on interview question form Face-to-face and audio-recorded interviews Telephone interview with or without audio- recording Diaries as basis for interviews Qualitative method

Focus groups interview Qualitative method Focus group interview –3-10 people –A group (or different groups) with a focus –May be from diverse user groups representative of target population –Requires a well-trained facilitator –Gather opinions, not behavior –Based on preset questions and agenda Facilitator can follow up on unanticipated issues, ideas, etc. –Interviews are recorded, transcribed Participants can be invited to listen to recording or read transcribed interview to explain comments or clarify certain responses –Can be expensive and time consuming

Interview questions Exploratory: Open questions –Typically unstructured questions Possible answers: closed questions –Typically structured questions Combination of both types

Guidelines for interview questions Ask clear, well-worded, non-leading questions Do not ask double barrel questions Ask questions that encourage participants to state opinion, express feelings, as applicable Pilot-test, refine, and use questions

Running the interview Introduce yourself Introduce purpose of interview –Participant to read and sign consent form –Mention time interview may take in consent form and verbally –Participant’s signature confirms willingness to participate Ask easy questions first such as demographic ones (warm-up session) –Opening questions Main questions Closed questions Thank you note

Observation Quantitative technique Gather information about behavior Provide information about users’ context, tasks, goals, success, difficulties, etc. Two types: –Direct –Indirect

Direct observation In a user’s field (natural setting) In a controlled environment (setting chosen by observer: lab, café, etc. ) In the field: Provides detail that can’t be captured through qualitative method Requires good planning: structure and focus –Think of who you are observing –What you are observing –Where the observation is taking place (get to know the environment) –See text, p. 325 for detailed framework.

Direct observation: Controlled environment Typically takes place in a usability lab. Can be performed in a setting using a portable usability lab and/or software to capture user actions, verbalization, and feelings Detail will be covered in Evaluation

Indirect observation Tracking users’ activities –Transaction logs –Diaries/journals Advantages/disadvantages of interaction logs Advantages/disadvantages of diaries/journals

Choosing and combining methods Variables to consider –Goal and focus of data gathering –Participants involved (who they are, how many, where they are, accessibility) –Time for data collection, transcription, coding, and analysis –Resources available –Other?

Survey and questionnaires Technique for collecting data and users’ opinions Used to capture data from group spread across a wide or small geographic area or place Level of success based on type and nature of questions asked

Questionnaires Questions can be like those used in structured or semi- structured interviews Closed questions Open-ended questions Combination of both types of questions

Questionnaire design Introduction Instructions on how to complete questionnaire Question sequencing Length of questionnaire Check boxes Scales (Likert; semantic differential scale) Yes, No, Don’t Know