1. Factorial Designs: Programmatic Research, RH: Testing & Causal Inference Applications of Factorial designs in Programmatic Research Research Hypotheses for Factorial Designs Variable Role Explication in Factorial Designs & Causal Interpretation

2. Using Factorial Designs in Programmatic Research I Adding a 2 nd IV Perhaps the most common application of factorial designs it so look at the separate (main) and combined (interaction) effects of two IVs Often our research starts with a simple RH: that requires only a simple 2-group BG research design. Tx1 Control Keep in mind that to run this study, we made sure that none of the participants had any other treatments !

3. At some point we are likely use Factorial designs to ask ourselves about how a 2 nd Tx/IV also relates to the DV Factorial Designs – Separate (Main) and combined (interaction) effects of two treatments Tx1 Control Tx2 Control Gets neither Tx1 not Tx2 Gets both Tx1 & Tx2 Gets Tx2 but not Tx1 Gets Tx1 but not Tx2

4. Using Factorial Designs in Programmatic Research II “Correcting” Bivariate Studies 40 Tx1 Tx2 40 Male Female Our well sampled, carefully measured, properly analyzed study showed … … nothing ! Our well sampled, randomly assigned, manipulated, controlled, carefully measured, properly analyzed study showed … … nothing ! Looks like neither TX nor Gender is related to the DV !!!

5. Tx 1 Tx 2 However, when we analyzed the same data including both variables as IVs … 60 2060 20 Male Female 40 There are treatment effects both for Males & Females – the marginal Tx means are an “aggregation error” So, instead of the “neither variable matters” bivariate results, the multivariate result shows that both variables are conditionally related to the DV -- they interact !!!!! There are Sex effects both for those in Tx1 & those in Tx2 – the marginal Sex means are an “aggregation error”

6. Using Factorial Designs in Programmatic Research III Generalization across Populations, Settings & Tasks Often our research starts with a simple RH: that requires only a simple 2-group BG research design. Computer Lecture Keep in mind that to run this study, we had to make some choices/selections: For example: population  College Students setting  Lecture setting stim/task  teach Psychology

7. When we’ve found and replicated an effect, making certain selections, it is important to check whether changing those selections changes the results. 6040 If there is an interaction – if the results “depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the “correct version” of the study to our theory or practice If there is no interaction – if the results “don’t depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the results of the study to our theory or practice, confident in their generalizability Computer Lecture

8. At some point we are likely use BG Factorial designs to ask ourselves how well the results will generalize to: other populations – college vs. high school Tx Control Col HS other settings – lecture vs. laboratory Tx Control Lecture On-line other tasks/stimuli – psyc vs. philosophy Tx Control Psyc Phil

9. Tx Control Col HS Tx Control Lecture On-line Tx Control Psyc Phil Notice that each factorial design includes a replication of the earlier design, which used the TX instructional methods to : teach Psychology to College Students in a Lecture setting Each factorial design also provides a test of the generalizability of the original findings: w/ Philosophy vs. Psychology to High School vs. College Students in an On-line vs. Lecture setting Tx Control 6040 6040 6040 6040 ??

10. Using Factorial Designs in Programmatic Research IV Do effects “depend upon” length of treatment ? As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design. Tx 1 Tx 2 Time Course Investigations In order to run this study we had to select ONE treatment duration (say 16 weeks): we assign participants to each condition begin treatment of the Tx groups treat for 16 weeks and then measured the DV 20

11. Using this simple BG design we can “not notice” some important things. A MG Factorial can help explore the time course of the Tx effects. Tx 1 Tx 2 Tx 1 Tx 2 Short Medium 20 40 By using a MG design, with different lengths of Tx as the 2 nd IV, we might find different patterns of data that we would give very different interpretations 20 Tx 1 Tx 2 Short Medium 20 40 Tx 1 Tx 2 Short Medium 20 60 40 Tx 1 Tx 2 Short Medium 20 40 0

12. Using Factorial Designs in Programmatic Research V Evaluating Initial Equivalence when Random assignment is not possible As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design. Tx 1 Tx 2 Initial Equivalence Investigations In order to causally interpret the results of this study, we’d have to have initial equivalence but we can’t always RA & manipulate the IV So what can we do to help interpret the post-treatment differences of the two treatments? Answer – compare the groups before treatment too!

13. Tx 1 Tx 2 Pre Post 20 40 By using a MG design, we can compare the groups pre-treatment and use that information to better evaluate post-treatment group differences (but can’t really infer cause). For which of these would you be more comfortable conclusing that Tx 1 > Tx 2 ?? Tx 1 Tx 2 Pre Post 40 20 40 Tx 1 Tx 2 Pre Post 60 20 40 Tx 1 Tx 2 Pre Post 30 2040 60 As good as it gets! Nah – Post dif = pre dif ! Nah – Tx 1 lowered score Maybe – more in- crease by Tx 1

14. Replication & Generalization in Factorial Designs Most factorial designs are an “expansion” or an extension of an earlier, simpler design, often by adding a second IV that “makes a variable out of an earlier constant”. This second IV may related to the population, setting or task/stimulus involved. Study #1 – Graphical software Study #2 Mean failures PC = 5.7, std = 2.1 Mean failures Mac = 3.6, std = 2.1 PC Mac Graphical Computing 5.9 3.1 4.5 3.6 3.8 3.7 What gives us the most direct replication? The main effect of PC vs. Mac or one of the SEs of PC vs. Mac? Did Study #2 replicate Study #1? Identifying the “replication” in a factorial design

15. Replication & Generalization in Factorial Designs, cont… Most factorial designs are an “expansion” or an extension of an earlier, simpler design, often by adding a second IV that “makes a variable out of an earlier constant”. This second IV may related to the population, setting or task/stimulus involved. Study #1 – Mix of Networked & Study #2 Stand-alone computers Mean failures PC = 5.7, std = 2.1 Mean failures Mac = 3.6, std = 2.1 PC Mac Networked Stand-alone 8.9 3.1 6.0 1.6 5.8 3.7 What gives us the most direct replication? The main effect of PC vs. Mac or one of the SEs of PC vs. Mac? Did Study #2 replicate Study #1? Identifying the “replication” in a factorial design

16. RH: for Factorial Designs Research hypotheses for factorial designs may include RH: for main effects involve the effects of one IV, while ignoring the other IV tested by comparing the appropriate marginal means RH: for interactions usually expressed as differences between hypothesized results for a set of simple effects tested by comparing the results of the appropriate set of simple effects That’s the hard part -- determining which set of simple effects gives the most direct test of the interaction RH:

17. #1 Sometimes the Interaction RH: is explicitly stated when that happens, one set of SEs will provide a direct test of the RH: (the other won’t) This is most directly tested by inspecting the simple effect of paper vs. computer presentation for easy tasks, and comparing it to the simple effect of paper vs. computer for hard tasks. Here’s an example: Easy tasks will be performed equally well using paper or computer presentation, however, hard tasks will be performed better using computer presentation than paper. Presentation Comp Paper Task Diff. Easy Hard = >

18. Your Turn... Young boys will rate playing with an electronic toy higher than playing with a puzzle, whereas young girls will have no difference in ratings given to the two types of toys. Type of Toy Elec. Puzzle Gender Boys Girls = > Judges will rate confessions as more convincing than do Lawyers, however, Lawyers will rate witnesses as more convincing than do Judges. Type of Evidence Confession Witness Rater Judge Lawyer <> SE Toy @ Gender SE Rater @ Evidence

19. #2 Sometimes the set of SEs to use is “inferred”... Often one of the IVs in the study was used in previous research, and the other is “new”. In this case, we will usually examine the simple effect of the “old” variable, at each level of the “new” variable this approach gives us a clear picture of the replication and generalization of the “old” IV’s effect. e.g., Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing. Let’s take this “apart” to determine which set of SEs to use to examine the pattern of the interaction...

20. Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing. Here’s the design and result of the earlier study about learning stats. Type of Instruction Comp Lecture > Here’s the design of the study being planned. Type of Instruction Comp Lecture Topic Stats Writing What cells are a replication of the earlier study ? So, which set of SEs will allow us to check if we got the replication, and then go on to see of we get the same results with the new topic ? Yep, SE of Type of Instruction, for each Topic...

21. Your turn.. I have previously demonstrated that rats learn Y-mazes faster than do hamsters. I wonder if the same is true for radial mazes ? I’ve discovered that Psyc majors learn statistics & Ethics about equally well. My next research project will also look at how well Sociology majors learn these topics. Type of Rodent Rat Hamster > Major Psyc Soc = Maze Y Radial ? > Type of Rodent Rat Hamster Topic Stats Ethics ? = Topic Stats Ethics SE Rodent @ Maze SE Topic @ Major

22. #3 Sometimes the RH: about the interaction and one about the main effects are “combined” this is particularly likely when the expected interaction pattern is of the > vs. > type (the most common pattern) Here’s an example… Group therapy tends to work better than individual therapy, although this effect is larger for patients with social anxiety than with agoraphobia. Type of Therapy Group Indiv. Anxiety Social Agora. > > Main effect RH: > Int. RH: So, we would examine the interaction by looking at the SEs of Type of Therapy for each type of Anxiety.

23. Young girls have better verbal skills than motor skills, however the difference gets smaller with age (DV = skill score) Type of Skill Verbal Motor Age 4 yrs 9 yrs > > Confession is considered more convincing than eyewitness testimony. This preference is stronger for jurors than judges. (DV = convincingness rating) Type of Evidence Confession Witness Rater Judge Jurors > > Your Turn… > > SE Skill @ Age SE Evidence @ Rater

24. Explicating Variable Roles Before a studyAfter the studyIVDV Potential Control Variables Confounds -- initial equiv. of subj vars -- ongoing equiv of proc vars Confound Variables -- subject var confounds -- procedural var confounds

25. Explicating Design Variables What I want you to be able to do is to tell the specific “function” of any variable in any study you read -- even if that variable is not mentioned in the description of the method & procedure ! We’ll start by reviewing basic elements of variables and functions Subject variables and procedural variables subject variables are things the value of which participants “bring with them” when they arrive at the study age, gender, personality characteristics, prior history, etc. Procedural variables are thing the value of which are “provided” or “created” by the researcher during the study

26. About Potential Confounding Variables Like IVs, potential confounds are causal variables they are variables that we think (fear) could have a causal influence on a subject’s DV score if equivalent (on the average) across IV conditions, then they are “control variables “ and contribute to the casual interpretability of the results if nonequivalent (on the average) across IV conditions, they are “confounds” that introduce alternative explanations of why the mean DV scores differed across the IV conditions Candidates for Confounding Variables variables that researchers in your area have attempted to control (recognized confounds) variables know to be causal influences upon your DV (previously effective IVs) that are not the IV in your study

27. Explicating the “role” of variables in research designs any given variable must be … a manipulated variable or a subject variable a DV or an IV or a control variable or a confound a control variable has either been... balanced (usually by RA or matching) or held constant or eliminated a confounding variable is either a problem with initial equiv. (subject variable) or ongoing equivalence (procedural variable) Remember: all subject variables are controlled by RA (of individuals) all subject variables are confounds in QE or NG designs (except for any that were used in post hoc matching) with a priori matching - all subject variables are controlled with post hoc matching -- only matching variable(s) is controlled

28. manipulated subject independent dependent confound controlled "constant" eliminated balanced matched random assignment Always pick ONE of these two !!! Always pick ONE of these four !!! If you say the variable is a CONTROL variable, always pick one of these three types of control !!! If you say the variable was controlled by BALANCING, be sure to tell which balancing technique was used If you say the variable was a CONFOUND, tell if confound of initial or ongoing equivalence

29. Now for the hard part… You have to do the variable role explication 3 (count ‘em 3) times  Once for each main effect and once for the interaction effect! For this to make more sense, let’s look at how you decide whether or not you can make a causal interpretation of an effect in a factorial design… Here’s the rule most folks carry in their heads (good rule!)… Start by assessing the causal interpretability of each main effect In order to causally interpret an interaction, you must be able to casually interpret BOTH main effects.

30. Study of Age and Genderno casually interpretable effects (main effects nor interaction) Study of Age and Type of Toy (RA + Manip) only casually interpretable effect would be the main effect of Type of Toy (not the main effect of Age, nor the interaction). Study Type of Toy (RA + Manip) and Playing Situation (RA + manip) all effects are causally interpreted (both main effects and the interaction). Some examples…

31. Here’s the real rule… You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects! Presentation Comp Paper Task Diff. Easy Hard = > 1 st - I can RA, manip & control Presentation – so I can causally interpret each of the simple effects of Presentation. 2 nd – I can RA, manip & control Task Difficulty – so I can causally interpret the difference between the causally interpretable simple effects of Presentation for Easy & Hard tasks! So I CAN causally interpret the interaction!!

32. Example #2 You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects! Presentation Comp Paper Gender Male Female = > 1 st - I can RA, manip & control Presentation – so I can causally interpret each of the simple effects of Presentation. 2 nd – I can’t RA, manip & control Gender – so I can’t causally interpret the difference between the causally interpretable simple effects of Presentation for Males & Females! So I CAN’T causally interpret the interaction!!

33. Example #3 You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects! Presentation Comp Paper Gender Male Female = > 2 nd - I can RA, manip & control Presentation – so I can causally interpret the difference between the Gender simple effects – even though I can’t causally interpret either of them! 1 st – I can’t RA, manip & control Gender – so I can’t causally interpret the difference between Males & Females during either presentation! So I CAN’T causally interpret the interaction!!

34. How does this apply to variable role explication?? Presentation Comp Paper Gender Male Female = > Take “motivation”… It is a subject variable. It is not the DV or an IV. It is not reasonably a constant. It is not a matching variable. There was RA to Presentation  so, motivation (and all other subject variables) is presumably controlled by balancing between computer and paper (for both males & females) There was not RA to Gender  so motivation (and all other subject variables) is presumably a confound (initial eq) when comparing computer and paper (for both males & females) Motivation would also be a confound for the interaction !!