Increasing your confidence that you really found what you think you found. Reliability and Validity

Instrument Reliability

Ways to Test for Reliability A test is reliable if it measures the same way over time (test-retest). Give the test once. Wait a few weeks and give it again to the same people. Do a correlation between the two tests.

Ways to Test for Reliability A test is reliable if random subsets of the test measure similarly (internal consistency). Split the test into two sub tests. Give the two tests to the same group. Do a correlation between the two tests. Or, to test a single instrument for internal consistency: – Cronbach’s alpha (more than two choice answers) – Kuder-Richardson (dichotomous answers)

Instrument Validity Suppose you ask a group of adults to rate the degree to which obfuscation is a primary component in improvisational terpsichory. You might get very reliable results on a test- retest correlation but it is unlikely that you are really measuring their perceptions of the issue.

Instrument Validity An instrument is valid when it actually measures what you want to measure. So, validity is context dependant, both in terms of content and sample. There are 5 ways to examine validity.

Instrument Validity An instrument is valid if experts agree that it is measuring what it is supposed to measure (content validity). Are all of the important parts of the concept measured represented in the measurement instrument?

Instrument Validity An instrument is valid if scores reflect what is known about a cohesive set of related knowledge (construct validity). Did the measures selected to assess teacher attitudes actually do it? Content validity after the fact.

Instrument Validity An instrument is valid if respondents believe that on the surface an instrument is measuring what it says it is measuring (face validity). Do respondents believe a question about purchasing habits seems like a valid way to assess educational philosophy? Also, does it look like instruments are supposed to look?

Instrument Validity An instrument is valid if there is a strong correlation with a known assessment of some construct (criterion-related validity). Build a survey about political conservatism and test it on people who voted republican. The predictive validity of the college SAT

Other Thoughts on Validity Data collection must be meaningful and appropriate to be valid. – Fair – Cost effective and efficient – Not harmful

Study (Not Instrument) Validity in Experimental Research Multiple plausible interpretations of the gathered data or: Questions about the ability to generalize to groups larger than the study sample Treats to internal validity Threats to external validity In experimental research, if studies are not designed correctly there may be:

Internal Validity in Experimental Research In an experiment you want to know that one thing (an intervention or an independent variable) caused something to happen. So, you have to design the study so that things other than the intervention don’t have an effect on the outcomes (dependent variables). There are some things that might allow for alternative explanations of the outcomes in experimental research—things that threaten the assumption that the independent variable is responsible for a change.

Threats to Internal Validity Why would you be concerned if the school started a DEAR program after you gave the pretest on a new reading curriculum? Outside reading may have an effect on how the students perform on the post-test, not the curriculum. History

Threats to Internal Validity Why would you be concerned if you were studying the impact of a conflict resolution curriculum over a year for 8 th grade students? 8 th graders go through huge social changes regardless of what you try to teach them. Maturation

Threats to Internal Validity Why would you be concerned if a number of kids who took the pre-test were not available for the post-test. Those kids might skew the results in important ways. Mortality or Attrition

Threats to Internal Validity Why would you be concerned if a pretest was multiple choice and the post-test was fill in the blank? The difference in scores might be based on the fact that some kids do better on one kind of a test than another. Instrumentation

Threats to Internal Validity Why would you be concerned if students had identical pre-test and post-test instruments? They may have improved because they remember how to complete the first test. Testing Instrumentation and Testing are conflicting threats.

Threats to Internal Validity Why should you be concerned if your data collection method was to set up a video camera in the back of a classroom to record a teaching event. The teacher and the students may be changing their behaviors because they know they are being recorded. Participant-Observer (like Instrumentation)

Threats to Internal Validity Regression to the mean Imagine a group of kids. What would the distribution of scores look like if they were all given the same test? Possible scores Our group

Threats to Internal Validity Regression to the mean Actually, it would look like this. This distribution occurs because of true scores and random error. Possible scores Our group

Threats to Internal Validity Pick the lowest kids The lower group is not a normal distribution Possible scores Our group Extreme Group Mean

Threats to Internal Validity Pick the lowest kids The lower group is not a normal distribution Possible scores Our group Extreme Group Mean

Threats to Internal Validity Regression to the mean Because of random error the group becomes more normal on retest Possible scores Extreme Group Mean Retest Group Mean

Threats to Internal Validity The retest of an extreme group will always produce a mean closer to the whole group Possible scores Our group Retest Group Mean

Threats to Internal Validity Regression to the mean appears at both ends of the distribution Be cautious of programs that target kids because of high or low scores Possible scores Our group Retest Group Mean

Threats to Internal Validity What is the best way to compensate for most threats to internal validity? Use a control group. If a group is treated just the same except they don’t get the intervention then the threats to internal validity are reduced.

Control Groups The problems with control groups are: – They have to be randomly selected to take advantage of inferential statistical procedure, or: – You have to demonstrate that the control group is not significantly different than the intervention group in any important way at the start of the study.

Control Groups DI to 4th Grade Class Post-Test Independent  Dependent Non-DI to different 4th Grade Class gain Compare  Pre-Test group data t-test

Control Groups DI to 4th Grade Class Post-Test Independent  Dependent Non-DI to different 4th Grade Class gain Compare  group data Pre-Test group data group data ANOVAt-test

Control Groups The biggest problem with control groups is that we can rarely use them in education. – True control groups get NO intervention. – What you are really doing is comparing interventions.

External Validity Threats to external validity (the ability to generalize to a larger group) are usually things that make the study not replicable in another setting (things that can’t be duplicated). – Experimenter affects results – Variables are too specific – Other interventions or assessments cause participants to respond irregularly

External Validity And the biggest threat to external validity of all in a statistical study is that the sample is not randomly selected.