Education 795 Class Notes Non-Experimental Designs ANCOVA Note set 5

Today’s Agenda Announcements (ours and yours) Q/A Non-experimental design Categorical predictors Statistical control ANCOVA Interactions

Nonexperimental Designs What distinguishes designs are a. manipulation of IV’s b. randomization Experiments have both a and b Quasi-experiments have a but not b Nonexperimental designs have neither a nor b

Prediction vs. Explanatory Predictive research is aimed at predicting outcomes from a collection of variable Explanatory research is aimed at testing hypotheses formulated to explain phenomena of interest Major difference is the role of theory. Explanatory research usually lacks a theoretical framework and it is a search to uncover independent variables

Logic of Comparison Experimental Designs comparison made among groups that, because of randomization, are equal across all things except for the “treatment” Quasi-experimental Designs comparison made among groups that have been exposed to different “treatments” but are groups that are often not like to begin with

Logic of Comparison Nonexperimental designs Often grouped based on the dependent variable… e.g. persisters/nonpersisters, users of technology, nonusers of technology… We then try to uncover the variables that have “caused” the observed differences in groups. So we have group problems: they are not alike across the independent variables they may have been exposed to the same treatment Sampling and design are key. In nonexperimental research we need to be very careful of the results and implications we put forth based on our samples and designs

Control vs. Comparison Groups The major threat to validity in nonexperimental research comes from uncontrolled unmeasured covariates… confounding variables

Categorical Independent Variables Used when we classify people into groups and are interested in group differences Unless a categorical variable is a “treatement”, there is no causation implied… in other words a difference between males and females on an outcome becomes a question, not an answer. What is it about males and females that make them differ on the phenomena being studied?

How We Treat Categorical IV’s For Continuous Outcomes Dichotomous independent variables t-tests Polychotomous independent variables ANOVA or multiple regression For nominal outcomes crosstabs, Chi-Square

Coding Categorical IV’s We are familiar with the dichotomous independent variable sex: female/male We have also been using race: students of color/white Reminder: the groups must be mutually exclusive We still stick to Dummy Coding in this class. (Behind the scenes, SPSS is doing effects coding)

Dummy Coding Consists of 1s and 0s with 1 signifying membership in a category and 0 signifying nonmembership Let’s extend our dichotomous IV to n levels. You will have n columns representing the n groups but will only use n-1 groups in a regression model.

Example With Data RELIGIONJCMO 1 Jewish Christian Muslim Other Muslim Jewish1000 Four columns represent four mutually exclusive groups. In this example, the first and sixth subjects are Jewish

Using Categorical IV’s In our example, we only need three of the columns to correctly specify a contrast between two groups The group you leave out of the regression, will become your “reference category” A reference category is technically represented by the constant Contrasts will be between the three included groups and the group you left out

Our Example Regression Dependent Variable: Promote Racial Understanding Independent Variable: Religion If I include C, M and O as variables and leave out J. Then I will have three variables representing three contrasts C-the difference between Christians and Jewish M-the difference between Muslims and Jewish O-the difference between Others and Jewish If we want a different set of contrasts we choose a different group as the referent group to leave out

Statistical Control Forms of control manipulation, elimination/inclusion, statistical, randomization Manipulation—the control the researcher has over manipulation of a treatment Elimination/Inclusion—either we eliminate by holding them constant (only study females) or we include them so they can be estimated Statistical—include them as covariates, control for them but not interested in them Randomization—with random assignment, we control for observed and unobserved covariates!!

Statistical Control in Our Context Rather than holding IVs constant through experimental control, influence is held constant by statistical techniques (by removing influence of confounding factors) Application to non-experimental designs makes causal interpretations difficult Measurement concerns are important for the IVs

ANCOVA The age old question: “What is the difference between ANCOVA and Regression?” Those trained in experimental research are usually taught to apply and “speak” in ANCOVA terms Those trained in quasi-experimental, correlational research are usually taught to apply and “speak” in Regression terms

ANCOVA They are parallel analytical techniques. One usually employs ANCOVA in cases where the “treatment” is manipulated and of a causal nature. The field of higher education primarily utilizes Multiple Regression. Doing ANCOVA through a multiple regression program not only enables one to see clearly what is taking place but also affords the control necessary to carry out the analysis required. (Pedhazur & Pedhazur, 1991, p. 568).

Theoretical: Interactions Without interactions between predictors in the model, we assume a constant effect for all levels of each independent predictor Interactions allow the effects of variables depend on the level of OTHER independent variables. Example: the effect of race for promoting racial understanding differs for genders implies an interaction between sex and race

Interactions Ordinal—The regression lines do not intersect within the range of another independent variable (the rank order of the effect does not change) Disordinal—The regression lines intersect within the range of another independent variable (the rank order of the effect changes, flips)

Graphical Interactions or Lack Of

ANCOVA Example Reminder: Including variables as statistical controls reduces the error variance thus increases the sensitivity of the analysis. Our example: dependent—promote racial understanding independents—attend cultural awareness workshop controls—sex, race interaction—race*cultural awareness workshop

Assumptions All the normal regression assumptions about normality, homogeneity, independence Assume effect of attending a cultural awareness workshop is constant across males and females Allow the effect of attending a cultural awareness workshop to vary across whites/students of color by adding the interaction

Results Note: 21% of students attended a workshop, attending at approximately equal rates across the race groups

Results

Intepretation The interaction is not significant so we can say: the effect of attending a cultural awareness workshop is constant across whites vs. students of color all three effects, sex, race, workshop are significant females, students of color and those attending workshops are more likely to believe promoting racial understanding is important

Graphing Interactions

Last But Not Least Adusted Means We do this by plugging values White, Male, Attend=1.99 White, Male, No Attend=1.41 SOC, Male, Attend=2.86 SOC, Male, No Attend=2.12

For Next Week Read Pedhazur Ch Read Pedhazur Ch 4 p66-70 Read Pedhazur Ch 22 p