1. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 1 INF 397C Introduction to Research in Library and Information Science Spring, 2005 Day 5

2. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 2 3 things today 1.Work the sample problems I handed out at the end of class last week. 2.Get serious about experimental design –How do we set up those IVs and DVs, how do we sample and set up groups? 3.Get some practice, as a group, actually designing an experiment.

3. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 3 Ch. 3 -- Ethics Read the chapter. Understand informed consent, p. 57 – a person’s expressed willingness to participate in a research project, based on a clear understanding of the nature of the research, the consequences of declining, and other factors that might influence the decision. Odd quote, p. 69 – Debriefing should be informal and indirect. Know that UT has an IRB: http://www.utexas.edu/research/rsc/humanresearch/

4. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 4 Ch. 7 – Independent Groups Design Description and Prediction are crucial to the scientific study of behavior, but they’re not sufficient for understanding the causes. We need to know WHY. Best way to answer this question is with the experimental method. “The special strength of the experimental method is that it is especially effective for establishing cause-and-effect relationships.”

5. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 5 Good Paragraph P. 196, para. 2 – Discusses how experimental methods and descriptive methods aren’t all THAT different – well, they’re different, but related. And often used together.

6. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 6 Good page – P. 197 Why we conduct experiments If results of an experiment (a well-run experiment!) are consistent with theory, we say we’ve supported the theory. (NOT that it is “right.”) Otherwise, we modify the theory. Testing hypotheses and revising theories based on the outcomes of experiments – the long process of science.

7. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 7 Logic of Experimental Research Researchers manipulate an independent variable in an experiment to observe the effect on behavior, as assessed by the dependent variable.

8. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 8 Independent Groups Design Each group represents a different condition as defined by the independent variable.

9. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 9 Random... Random Selection vs. Random Assignment –Random Selection = every member of the population has an equal chance of being selected for the sample. –Random Assignment = every member of the sample (however chosen) has an equal chance of being placed in the experimental group or the control group. Random assignment allows for individual differences among test participants to be averaged out.

10. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 10 Let’s step back a minute An experiment is personkind’s way of asking nature a question. I want to know if one variable (factor, event, thing) has an effect on another variable – does the IV systematically influence the DV? I manipulate some variables (IVs), control other variables, and count on random selection to wash out the effects of all the rest of the variables.

11. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 11 Block Randomization Another way to wash-out error variance. Assign subjects to blocks of subjects, and have whole blocks see certain conditions. (Very squirrelly description in the book.)

12. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 12 Challenges to Internal Validity Testing intact groups. (Why is the group a group? Might be some systematic differences.) Extraneous variables. (Balance ‘em.) (E.g., experimenter). Subject loss –Mechanical loss, OK. –Select loss, not OK. Demand characteristics (cues and other info participants pick up on) – use a placebo, and double- blind procedure Experimenter effects – use double-blind procedure

13. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 13 Role of Data Analysis in Exps. Primary goal of data analysis is to determine if our observations support a claim about behavior. Is that difference really different? We want to draw conclusions about populations, not just the sample. Two different ways – statistics and replication.

14. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 14 Two methods of making inferences via stats Null hypothesis testing –Assume IV has no effect on DV; differences we obtain are just by chance (error variance) –If the difference is unlikely enough to happen by chance (and “enough” tends to be p <.05), then we say there’s a true difference. Confidence intervals –We compute a confidence interval for the “true” population mean, from sample data. (95% level, usually.) –If two groups’ confidence intervals don’t overlap, we say (we INFER) there’s a true difference.

15. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 15 What data can’t tell us Proper use of inferential statistics is NOT the whole answer. –Scientist could have done a trivial experiment. –Also, study could have been confounded. –Also, could by chance find this difference. (Type I and Type II errors – hit this for real in week 6.)

16. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 16 This is HUGE. When we get a NONsignificant difference, or when the confidence intervals DO overlap, we do NOT say that we ACCEPT the null hypothesis. –Hinton, p. 37 – “On this evidence I accept the null hypothesis and say that we have not found evidence to support Peter’s view of hothousing.” We just cannot reject it at this time. We have insufficient evidence to infer an effect of the IV on the DV.

17. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 17 Notice Many things influence how easy or hard it is to discover a difference. –How big the real difference is. –How much variability there is in the population distribution(s). –How much error variance there is. –Let’s talk about variance.

18. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 18 Sources of variance Systematic vs. Error –Real differences –Error variance What would happen to the standard deviation if our measurement apparatus was a little inconsistent? There are OTHER sources of error variance, and the whole point of experimental design is to try to minimize ‘em. Get this: The more error variance, the harder for real differences to “shine through.”

19. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 19 One way to reduce the error variance Matched groups design –If there’s some variable that you think MIGHT cause some variance, –Pre-test subjects on some matching test that equates the groups on a dimension that is relevant to the outcome of the experiment. (Must have a good matching test.) –Then assign matched groups. This way the groups will be similar on this one important variable. –STILL use random assignment to the groups. –Good when there are a small number of possible test subjects.

20. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 20 Another design Natural Groups design –Based on subject (or individual differences) variables. –Selected, not manipulated. –Remember: This will give us description, and prediction, but not understanding (cause and effect).

21. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 21 We’ve been talking about... Making two groups comparable, so that the ONLY systematic difference is the IV. –CONTROL some variables. –Match on some. –Use random selection to wash out the effects of the others. –What would be the best possible match for one subject, or one group of subjects?

22. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 22 Themselves! When each test subject is his/her own control, then that’s called a –Repeated measures design, or a –Within-subjects design. (And the independent groups design is called a “between subjects” design.)

23. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 23 Repeated Measures If each subject serves as his/her own control, then we don’t have to worry about individual differences, across experimental and control conditions. EXCEPT for newly introduced sources of variance – order effects: –Practice effects –Fatigue effects

24. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 24 Counterbalancing ABBA Used to overcome order effects. Assumes practice/fatigue effects are linear. Some incomplete counterbalancing ideas are offered in the text.

25. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 25 Which method when? Some questions DO lend themselves to repeated measures (within-subjects) design –Can people read faster in condition A or condition B? –Is memorability improved if words are grouped in this way or that? Some questions do NOT lend themselves to repeated measures design –Do these instructions help people solve a particular puzzle? –Does this drug reduce cholesterol?

26. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 26 Hinton typo P. 62, para. 1: “... population standard deviation, µ, divided by....”

27. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 27 Midterm Emphasize –How to lie with statistics – concepts –To know a fly – concepts –SZ&Z – Ch. 1, 2, 7, 8 –Hinton – Ch. 1, 2, 3, 4, 5 De-emphasize –SZ&Z – Ch. 3 –Other readings Totally ignore for now –SZ&Z – Ch. 14 –Hinton – Ch. 6, 7, 8

28. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 28 Some questions we’d like to ask Nature

29. R. G. Bias | School of Information | SZB 562BB | Phone: 512 471 7046 | rbias@ischool.utexas.edu i 29 Next Week NO new stuff. All review. We’ll work some stat problems in class. We’ll design another experiment or two. I’ll respond to questions. Attendance voluntary but recommended.