Analysis of Complex Designs Steps for Data Analysis: Check the data for errors and outliers. Summarize the results using descriptive statistics. The factorial design tells you how many means you need to analyze. 2 x 2 design has 2 main effects each with 2 marginal means each 4 conditions, so 4 cell means Graph the cell means “means for each condition” Confirm what the data reveal. Use inferential statistics to determine if means are different simply because of chance (error variation), or whether the independent variables caused the means to differ.

Table 8.4 Guidelines for Analysis of Two-Factor Experiment Is the A X B interaction effect statistically significant? No Yes Are the main effects Are the simple main significant? effects significant? No Yes No Yes Compare Compare Stop two means Stop two means

Analysis of Complex Designs Effort to Obtain a Confession Guidelines for the analysis of a complex design First, determine whether the interaction effect between the independent variables is statistically significant. See fig 8.1 below If so, identify the source of the interaction effect by examining the simple main effects Pair-wise differences between the cell means “conditions” at each level of the IV

Analysis of Complex Designs Effort to Obtain a Confession Guidelines for the analysis of a complex design Simple main effect of Suspect Status At Guilty expectation level 5.64 to 7.17 is significant At Innocent expectation level 5.56 to 5.85 not significant Simple main effect of Interrogator Expectation For Actual Guilty level comparing 5.64 to 5.56 For Actual Innocence level comparing 7.17 to 5.85 Note: do not need to use both of these

Analysis Plan with No Interaction Effect If the analysis indicates the interaction effect is not statistically significant. The next step is to determine whether the main effects of the variables are statistically significant Significant main effects can be determined by comparing means for each level of the IV or by using confidence intervals to compare those means

Table 8.4 Guidelines for Analysis of Two-Factor Experiment Is the A X B interaction effect statistically significant? No Yes Are the main effects Are the simple main significant? effects significant? No Yes No Yes Compare Compare Stop two means Stop two means

Analysis Plan with No Interaction Effect Interrogator Expectation Guilt (3.62) compared to Innocent (2.60) This main effect of Interrogator Expectation maybe statistically significant Suspect Status Actual Guilt (3.04) compared to Actual Innocence (3.18) This main effect of Suspect Status probably not statistically significant Means for Interrogator Expectation: 3.62 2.60

Analysis Plan with an Interaction Effect Kaiser 2006 social identity threat research using the emotional Stroop task when words are presented subliminally has a significant interaction Using a 2 x 3 design Social identity (random groups design) Threat: male partner with sexist views Safety: male partner without sexist views Word type (repeated measures design) Social identity threat; sexist words Illness threatening; disease words Nonthreatening; household object words Response time to identify the color of the word

Analysis Plan with an Interaction Effect Kaiser 2006 social identity threat research using the emotional Stroop task when words are presented subliminally there was a significant interaction between Independent Variables In a 2 X 3 design the interaction can be at just one level of an I.V. Because there is an interaction identify the source of the interaction effect by examining the simple main effects

Analysis Plan with an Interaction Effect Guidelines for the analysis of a complex design Compare simple main effects at each level of the I.V. Do not need to use all possible simple main effects Threat vs Safety at the Social-identity Threatening level 598.9 vs 603.9 Threat vs Safety at the Illness Threatening level 577.7 vs 615.0 Threat vs Safety at the Non-Threatening level 583.9 vs 614.5 Social-identity vs illness vs non for the threatening level 598.9 vs 577.7 vs 583.9 are significantly different Follow up on significant differences by examining the simple main effect When a simple main effect with three or more levels is statistically significant, conduct comparisons of means two at a time. 598.9 vs 577.7, 598.9 vs 583.9, 577.7 vs 583.9 Social-identity vs illness vs non for the Safety level 603.9 vs 615.0 vs 614.5 are not significantly different Do not need to follow up

Analysis Plan with No Interaction Effect For example Kaiser 2006 when words can be processed consciously Mean response times No Interaction between I.V. Analyze main effects Social Identity 613.6 vs 631.4 Word type is significant 637.5 vs 617.6, 610.8 Follow up by comparing means Two at a time

Interaction Effects and Theory Testing Theory will predict interaction of variables Need a complex design to test the theory Kaiser et al. (2006) social identity theory tested hypotheses about attention to prejudice cues in the environment From prior research when individuals’ social identity is threatened they are consciously aware of cues in their environment relating to potential prejudice extend this research Do threatened individuals pay attention to prejudice cues nonconsciously, without awareness? Complex designs can resolve differences between studies Combine I.V. from contradictory studies in a complex design

Interaction Effects and External Validity Interaction effect is not present Generalize findings across conditions of experiment Interaction effect is present Sets limits on generalizing a finding Conditions of experiment identify the limits

Interaction Effects and External Validity The presence of a statistically significant interaction effect sets limits on the external validity of a finding. For example, in the Kassin et al. (2003) study Interaction of interrogator expectation and suspect status Combination of Guilty level of interrogator expectation actual innocence level of the suspect status produces the highest score on effort to obtain confessions So we cannot generalize findings for interrogator expectation It has an influence when there is actual innocence It does not when there is actual guilt

Interaction Effects and External Validity Because Kassin et al. (2003) observed a statistically significant interaction effect for Interrogator expectations and Suspect Status Interrogator expectations depends on the suspect status of an individual There are limits to external validity

Interaction Effects and External Validity When an interaction effect is not statistically significant in an experiment, the researcher can generalize the findings across the conditions of the experiment. To understand this, we can look at Kassin et al. (2003) Number of Guilt-Presumptive Questions variable Main effect of interrogator expectation with guilty greater then innocent

Interaction Effects and External Validity More guilt-presumptive questions for guilty compared to innocent This was true for actual guilt and actual innocent therefore, the effect of suspect status generalized across the levels of interrogator expectations

Interaction Effects and External Validity Floor and Ceiling Effects Sometimes an interaction effect can be statistically significant “by mistake.” This occurs when: the means for one or more condition reach the highest possible score (ceiling effect) the lowest possible score (floor effect). When floor or ceiling effects occur, an interaction effect is uninterpretable.

Ceiling Effect: Example This graphs shows an interaction effect between Test Difficulty (easy, hard) and Study Hours (10, 15). Hours of study had an effect only in the hard-test condition, not in the easy-test condition. How do we interpret this interaction effect when we know the highest possible score on the tests is 50?

Ceiling Effect If we have enough “room” in our dependent variable to assess the effect of the independent variables, the interaction effect disappears. This graph shows two main effects: A main effect of Study Hours and a main effect of Test Difficulty.

Interaction Effects and Natural Groups Designs Using complex designs, researchers can test causal inferences for natural groups variables. Recall that we can’t make causal inferences with natural groups variables. Natural groups variables are correlational. So, how can we make causal inferences using a complex design?

Interaction Effects and Natural Groups Designs Steps for making causal inferences about natural groups variables in a complex design: State your theory. Why are the groups different? What is the theoretical process? Identify a relevant independent variable. This IV should influence the likelihood that the theorized process will occur (e.g., relationship maintenance). Look for an interaction effect. In order to make a causal inference, the natural groups variable and manipulated variable should produce a statistically significant interaction effect in the predicted direction. This interaction effect allows us to make causal inferences about why individuals differ — that is, we begin to understand why people differ.

Interaction Effects and Natural Groups Designs We can make causal inferences about natural groups when we test a theory for why the natural groups differ. For example, we can theorize that musicians and nonmusicians different in musical performance because of the way that these groups organize melodies By manipulating type of musical structure Test for interaction