Information Gathering Using Quantitative Methods Peter Lyman IS 208 Information Management SIMS February 8, 2001

Today’s topics 1. Research design. 2. Building a survey instrument. 3. Implementing an interview protocol.

ASSIGNMENT Create a research design for your class project. Define objectives Design instrument Create protocol Test design on a classmate.

Quantitative research design 1. Defining your goal & objective. 2. Choosing a method: quantitative or qualitative? 3. Building a survey instrument. 4. Pre-testing the instrument. Defining an interviewing protocol. Interviewing 7. Coding the results 8. Statistical Analysis 9. Dissemination of the findings

1. Defining the objective How will new information add value? for whom? when? how? Market research Formative evaluation = feedback for a project in progress Final assessment = measuring the outcomes of a project Longitudinal studies

Goals and objectives… When you gather information, you also are sending a message to your respondents. Your questions inform people about an issue Responding to a questionnaire is a form of participation Your findings change the perception of a project.

2. quantitative or qualitative? Types of methods. Questionnaires Structured Interviews Content analysis of documents and records Metric analysis of videotape Focus Groups Open ended Interviews Oral histories Participant Observation Ethnography

2. quantitative or qualitative? Costs and benefits numeric breadth structured data replicable tends to be faster good for defining scope and range of opinions textual depth defines cultural nuance and intensity of feelings case studies tends to take time good for identifying possible survey questions & language

Quantitative or qualitative? Research designs often combine methods Ethnography identifies issues and language for survey research Focus group themes often probe survey research findings Assessment combines structured surveys and open ended interviews

3. Creating a survey instrument Word the questions to avoid ambiguity and anticipate cultural differences in meaning. Question sequence should be logical. Question formats can elicit more/less complex information (T/F, Lickert scales, etc) Make sure responses create clear numeric codes.

4. Pre-testing Does the wording make sense to different kinds of respondents? Is it hard to administer the instrument? Is the interviewer able to get people committed to taking the questionnaire? Is the interviewer professional?

5. The survey protocol/setting How will the survey be implemented? Face to face interviews phone interviews computer surveys mail surveys ISSUES: Response rate, suitability of instrument, cost.

5. protocol /record keeping What kind of record keeping is needed? If open ended, write down responses audio and videotape may add value, but increase respondent resistance, and add cost in coding and interpretation.

5. protocol/sample Sample statistical validity depends upon sample size statistical validity depends upon random sample (Accept no substitutes, follow up to catch right subject is your sample motivated to take your survey?

6. Interviewing/consent and motivation An introduction should quickly describe who is doing the research and why, to overcome privacy concerns and engage the respondent. Federally sponsored/University research must be certified to meet “informed consent” standards.

6. Interviewing: establish a relationship Engage the subject, but stay professional Never deviate from script Do not interrupt learn to listen actively No personal views Don’t interpret Don’t improvise These are sources of error. Have a protocol to answer questions!!

6. Interviewing/notations After the interview, note the quality of the relationship, interview, and any possible error (e.g., interruptions, sample problems, social pressure). Take notes immediately, no longer than 24 hours.

7. Coding Structured interview requires simple, often machine scoring. Open ended questions require accurate (often double blind) coding following pre-defined categories.

8. Analysis Statistical analysis is coin of the realm. Learn to listen to the data, to see the unexpected. Learn to contemplate the data to figure out what it is saying. Data visualization (graphics) helps you to recognize what it is saying.

9. Dissemination Executive summary = what are the headlines? Graphics often tell the story best. In formative evaluation, it’s often necessary to show the data, to get people to listen. In assessment people often don’t want to hear bad news.

Frequent sources of error Pleasing/deceiving the interviewer Wrong type of interview (e.g., phone call during dinner) Flawed interviewing technique Sample Not listening to the data

Next time. Qualitative methods. How to run and interpret a focus group. In the readings, focus on: understanding group process issues (how are these reflected in group projects in SIMS?) What are key focus group techniques What you can/cannot learn from focus groups.