Group Experimental Design

Validity Validity is the degree to which the data are both meaningful and adequately represent the construct under study.  Content validity is the extent to which the measured variables represent all the possible measures of content or skills.  In surveys, do the questions represent all the possible questions that could be asked?  With respect to performance, does the particular measure, e.g., attendance or computation, represent all possible components of the construct, e.g.,  parent participation or mathematics

Relationship of Reliability to Validity Cannot have validity without reliability A poorly worded question on a survey cannot be valid because it is not reliable, It does not consistently assess participants’ attitude because it will be interpreted differently by the different participants.  A well-worded question could be reliable but still may not be valid It may be unrelated or tangentially related to the construct under study.  Similarly, poorly worded behavioral definitions will not yield reliable data

Can have reliability without validity Well-defined definitions will yield reliable data But such data still may not meaningfully or completely represent the target construct, attendance does not adequately represent parent participation

Table 6. 4 p. 172 Form of Validity What is for? How do we use it? What else do I need know? Content Validity Criteria Related Validity Construct Validity

Unit of Analysis The unit of analysis is the level of analysis (student, family, school, community, organization) by which the data need to be collected and analyzed to answer the questions asked. We must not only consider the questions, but also moderating and mediating variables implicit in our questions.

Sampling Representative of a given population Needs to be some target population

Probabilistic vs. Non-probabilistic Sampling Quantitative Sampling Probability Sampling Simple Random Stratified Multi-stage cluster Non-probability sampling Convenience Snowball

Experimental Design The purpose of an intervention (treatment or experimental) design is To determine the effect of the intervention (the independent or manipulated variables) on the outcomes (the dependent variables) To show causality: A caused X to occur

Experimental Design To control the effect of all other potential influencing variables, except for the particular intervention To show that it was the intervention alone that causes the outcome Potential extraneous (“other”) influencing variables The participants The treatments The procedures To reduce threats to internal validity

What is Internal Validity? Threat to Internal Validity What is it? Participants threat History Maturation Regression Selection Mortality Interactions with selection

What could you do about it? What do I need know? Treatment threats What is it? What could you do about it? What do I need know? Diffusion of Treatments Compensatory equalization Compensatory rivalry Resentful demoralization Procedure Threats Testing Instrumentation

External Validity: Insuring it applies to other situations Interaction selection and treatment Interaction of setting and treatment Interaction of history and treatment

What is common? What is not? What are challenges with each design? True Experimental Post test Experimental Quasi-Experimental –Pre-Post Quasi- experimental Post-test only

Between Groups Design

Factorial Design (2 x 3) Design a 3 X 3 X 4 Multiple Levels – we often call these grouping variables in data sets. PPT Clicker 5 Technology Instruction type Typical At-risk Special ED Risk Level

Main Effects vs. Interaction Effects Main Effect-Overall influence of the independent variable Interaction Effect- when the influence on one independent variable covaries (depends on) the other independent variable Look at p. 318 What does it mean to have an interaction effect?

REPEATED MEASURE DESIGN Time 1 Time 2 Time 3