ED 571 Introduction to quantitative research Virtual Meeting 3

Welcome  Course Instructor: Dr. Allen Kitchel Ensure everyone is in ZOOM and can virtually raise their hand. Provide general overview of ZOOM Session Always a bit of an experiment ... Will record sessions. Use Microphone like a Walky-Talky. Turn on and off as you use it. Turn Video camera’s on (unless this causes bandwidth problems). What is our plan if the technology fails? The Wheel (method used sometimes for asking questions of the group) Instructor has option to remove participants if something has gone wrong (e.g. left microphone open, too many video camera's on, etc.). If you are removed, just log back in and re-enter the meeting.

Normal Distribution: And finally, even more …

Virtual Meeting 2 Summary Review primary concepts from VM2 Normal Distribution, measurement validity and reliability, confidence intervals, sample size calculations, requirements for cause & effect, pre-experimental design, experimental design, quasi-experimental design, design notation, internal validity considerations Today: Virtual Meeting 3 Factorial designs Block designs Expanded discussion about quasi-experimental designs Introduction to inferential statistical analysis using Excel (t tests) Review the criteria for the “Research Proposal” Overview between now and end of semester Discussion/questions

Design Notation Post-Test Only Experimental Design Examples Review: Two-group Post-test Only Design – Sketch Student Groups: (a) Introductions & share research interest, (b) What would you measure for your research area?, (b) How would the groups be created?, (c) Would these be treatment vs. control, or treatment 1 vs treatment 2? (d) Would you need more than two groups? Whole group sharing

Design Notation Post-Test Only Experimental Design Examples Review: Two-group Post-test Only Design – Sketch Student Groups: (a) Introductions & share research interest, (b) What would you measure for your research area?, (b) How would the groups be created?, (c) Would these be treatment vs. control, or treatment 1 vs treatment 2? (d) Would you need more than two groups? Whole group sharing Student Groups: Consider confounding variables. What other variables might effect the measures. Are there things about demographic or background characteristics that might change the measures based on these groupings? Are there other ways to implement the treatment that need to be taken into account, for example, perhaps its not just the treatment, but the treatment combined with the treatment setting that may effect measures.

Factorial Design represented in Table Format Student example? Factor 1 (indicate levels) and factor 2 (Indicate levels) Factor 1 F1 L1 F1 L2 Factor 2 F2 L1 Measure

Factorial Design 2 x 2 Factorial Design – see next slides Allows one to examine “levels” of the variables (factors), and, most importantly, examine interactions between factor levels. Interaction effects most important, but main effects also considered. A researcher would do this based on theory, or to confirm whether or not there is a confounding variable that needs to be accounted for. Let’s use the example from the book: Factor 1 = Setting (In-Class vs. Pull-out) Factor 2 = Time in Instruction (1 hour vs. 4 hour) – see next slides

Factorial Design represented in Table Format Let’s use the example from the book: Factor 1: Time in Instruction (1 hour vs. 4 hour) Factor 2: Setting (In-Class vs. Pull-out) Factor 1 F1 L1 F1 L2 Factor 2 F2 L1 Measure

9.4a The Basic 2 x 2 Factorial Design Figure 9.9 An example of a basic 2 3 2 factorial design. Factor: A major independent variable. Level: A subdivision of a factor into components or features. Figure 9.10 Design notation for a 2 x 2 factorial design

9.4a The Basic 2 x 2 Factorial Design: Possible Outcome Figure 9.11 The null effects case in a 2 x 2 factorial design.

9.4a A Main Effect of Time in Instruction in a 2 x 2 Factorial design Figure 9.12 A main effect of time in instruction in a 2 x 2 factorial design.

9.4a A Main Effect of Setting in a 2 3 2 Factorial Design Figure 9.13 A main effect of setting in a 2 x 2 factorial design.

9.4a Main Effects of Both Time and Setting in a 2 x 2 Factorial Design Figure 9.14 Main effects of both time and setting in a 2 x 2 factorial design.

9.4a An Interaction in a 2 x 2 Factorial Design Figure 9.15 An interaction in a 2 x 2 factorial design

9.4a A Crossover Interaction in a 2 x 2 Factorial Design Figure 9.16 A crossover interaction in a 2 x 2 factorial design.

9.4b Benefits and Limitations of Factorial Designs Enhances the signal Efficient design Only design that allows you to examine interactions Limitations Complex More participants required

9.4c Factorial Design Variations: 2 x 3 Figure 9.17 Main effect of setting in a 2 x 3 factorial design.

9.5 Noise-Reducing Designs: Randomized Block Designs Helps minimize noise through the grouping of units (e.g., participants) into one or more classifications (blocks) that account for some of the variability in the outcome Beneficial when theorized groups have homogenous characteristics towards the measure. A form of analysis, does not require changing setup of experimental. Design notation indicates blocks of analysis Examples: Grade level; Testing of tires Figure 9.24 The basic randomized block design with four groups or blocks.

9.8 Limitations of Experimental Design Differential drop out (mortality threat) Ethical problems Social threats to internal validity Difficult to generalize to the real world

Discuss and Debate What is the difference between random selection and random assignment? What are some strengths and weaknesses of experimental designs? Can you think of some research topics for which a factorial design may be a good approach? Need at least two variables with multiple levels. Random selection is the process of randomly drawing a sample from the population. Random assignment, on the other hand, involves taking the sample the researcher has drawn, and then randomly assigning each unit (participant) to either the treatment group or the control group.   Experiments are often considered to be “the gold standard” in social science research, but they are also criticized for being artificial. What is observed under tightly controlled conditions may not be what happens in the natural environment. Experiments also bring up a host of ethical concerns, which must be carefully addressed. Other research designs may be appropriate, such as a quasi-experimental design, or a qualitative approach. Any time the researcher wishes to enhance the “signal” in a construct, a factorial design is a good strategy to use. In addition, factorial designs allow researchers to expand the number of independent variables in a study, as well as add levels to those variables. Ask your students to provide some real-world examples (e.g., spending $100 or $200 at the mall versus online).

10.1 Foundations of Quasi-Experimental Design “Quasi” means “sort of”, Quasi-experiments have: A control group A treatment (or program) group Variables Quasi-experiments do not have: Random assignment to groups

10-2 The Nonequivalent Groups Design One of the most frequently used quasi-experimental designs Looks just like a pretest-posttest design Lacks random assignment to groups As a result, the treatment and control groups may be different at the study’s start Raises a selection threat to internal validity Figure 10.2 Notation for the Nonequivalent-Groups Design (NEGD).

10.2a Plot of Pretest & Posttest Means for Possible Outcome 1 Figure 10.6 Plot of pretest and posttest means for possible outcome. Here, there are selection threats. Selection threats are any factor other than the program that leads to posttest differences between groups. The comparison group started with a lower score on the pretest than the program group, so the groups were not equivalent from the start. The comparison group did not change from pretest to posttest, while the program group did—they improved. This could be due to selection-history effects, selection-maturation effects, or selection-regression effects.

10.2a Plot of Pretest & Posttest Means for Possible Outcome 2 Figure 10.7 Plot of pretest and posttest means for possible outcome 2. Here, both the program and the comparison group improve from pretest to posttest, but there are still differences between the groups from the start of the study. Selection-testing, selection-instrumentation, and selection-mortality are all plausible explanations for this outcome.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 3 Figure 10.8 Plot of pretest and posttest means for possible outcome 3 Here, a selection-regression effect is the most likely explanation for the results. A selection-regression threat is a threat to internal validity that occurs when there are different rates of regression to the mean in the two groups. The comparison group starts lower than the program group, and does not change throughout the study. The program group starts out very high on the pretest, and there is a sharp drop in the group’s performance, which is likely due to regression toward the mean.

10.2a Plot of Pretest and Posttest Means for Possible Outcome 4 Figure 10.9 Plot of pretest and posttest means for possible outcome 4. Here, a selection-regression threat is also a likely explanation. The program group starts out considerably lower than the comparison group, and shows a sharp increase in performance. The comparison group does not change. This is likely because the program group is regressing up toward the mean.

10.2a Plot of Pretest & Posttest Means for Possible Outcome 5 Figure 10.10 Plot of pretest and posttest means for possible outcome 5. Here, most threats to internal validity are unlikely, and the treatment is likely causing the result.

10.3 The Regression-Discontinuity Design A pretest- posttest program comparison- group quasi-experimental design in which a cutoff criterion on the preprogram measure is the method of assignment to a group

10.3a The Basic RD Design Notation C indicates that groups are assigned by means of a cutoff score on the premeasure An O stands for the administration of a measure to a group. An X depicts the implementation of a program Each group is described on a single line Figure 10.11 Notation for the Regression-Discontinuity (RD) design.

10.3a Regression Line A line that describes the relationship between two or more variables Figure 10.12 Pre-post distribution for an RD design with no treatment effect. Figure 10.13 The RD design with ten-point treatment effect.

10.4a The Proxy Pretest Design A post-only design in which, after the fact, a pretest measure is constructed from preexisting data Usually done to make up for the fact that the research did not include a true pretest Figure 10.16 The Proxy- Pretest design.

10.4b The Separate Pre-Post Samples Design A design in which the people who receive the pretest are not the same as the people who take the posttest Figure 10.17 The Separate Pre-Post Samples design.

10.4c The Double-Pretest Design A design that includes two waves of measurement prior to the program Addresses selection-maturation threats Figure 10.19 The Double-Pretest design.

10.4d The Switching-Replications Design A two-group design in two phases defined by three waves of measurement In the repetition of the treatment, the two groups switch roles: The original control group in phase 1 becomes the treatment group in phase 2, whereas the original treatment group acts as the control Figure 10.20 The Switching-Replications design.

10.4e The Nonequivalent Dependent Variables (NEDV) Design At first, looks like a weak design But pattern matching gives researchers a powerful tool for assessing causality The degree of correspondence between two data items Figure 10.21 The NEDV design. Figure 10.23 Example of a Pattern-Matching variation of the NEDV design.

10.4f The Regression Point Displacement (RPD) Design A pre-post quasi-experimental research design where the treatment is given to only one unit in the sample, with all remaining units acting as controls This design is particularly useful to study the effects of community level interventions Figure 10.24 The Regression Point Displacement (RPD) design

Discuss and Debate Why can quasi-experiments be more ethical than randomized experiments? What are the strengths and the weaknesses of quasi-experimental designs? In a quasi-experiment, those participants who may need a treatment or program the most can be included in the experimental group, whereas in a randomized experiment, they cannot.   Quasi-experimental designs can be preferable to randomized experiments for ethical reasons and also for logistical reasons (sometimes, randomization is not possible due to the research topic or accessible population). However, quasi-experiments have internal validity problems that must be addressed in both the design and the analysis of the study. Sometimes, these designs require more participants than a randomized study, in order to increase statistical power.

Statistical Analysis of the Difference Between Groups (t Tests) See provided Excel data file for example data. Review example file and conduct basic analysis. Comparison of group means to determine if there is a “statistically significant difference” between groups. Do you know what it means when we say “statistically significant difference”? Hint – consider issues of sampling error.