1. Mark W. Lipsey Vanderbilt University
Specifying the Conceptual and Operational Models and the Research Questions that Follow
Mark W. Lipsey
Vanderbilt University
IES/NCER Summer Research Training Institute, 2010

2. Focus on randomized controlled trials
Purpose of the Summer Training Institute: Increasing capacity to develop and conduct rigorous evaluations of the effectiveness of education interventions
Caveat: “Rigorous evaluations” are not appropriate for every intervention or every research project involving an intervention
They require special resources (funding, amenable circumstances, expertise, time)
They can produce misleading or uninformative results if not done well
The preconditions for making them meaningful may not be met.

3. Critical preconditions for rigorous evaluation
A well-specified, fully developed intervention with useful scope
basis in theory and prior research
identified target population
specification of intended outcomes/effects
“theory of change” explication of what it does and why it should have the intended effects for the intended population
operators’ manual: complete instructions for implementing
ready-to-go materials, training procedures, software, etc.

4. Critical preconditions for rigorous evaluation (continued)
A plausible rationale that the intervention is needed; reason to believe it has advantages over what’s currently proven and available
Clarity about the relevant counterfactual– what it is supposed to be better than
Demonstrated “implementability”– can be implemented well enough in practice to plausibly have effects
Some evidence that it can produce the intended effects albeit short of standards for rigorous evaluation

5. Critical preconditions for rigorous evaluation (continued)
Amenable research sites and circumstances:
cooperative schools, teachers, parents, and administrators willing to participate
student sample appropriate in terms of representativeness and size for showing educationally meaningful effects
access to students (e.g., for testing), records, classrooms (e.g., for observations)

6. IES funding categories
Goal 2 (intervention development) for advancing intervention concepts to the point where rigorous evaluation of its effects may be justified
Goal 3 (efficacy studies) for determining whether an intervention can produce worthwhile effects; RCT evaluations preferred.
Goal 4 (effectiveness studies) for investigating the effects of an intervention implemented under realistic conditions at scale; RCT evaluations preferred.

7. Specifying the theory of change embodied in the intervention
Nature of the need addressed
what and for whom (e.g., 2nd grade students who don’t read well)
why (e.g., poor decoding skills, limited vocabulary)
where the issues addressed fit in the developmental progression (e.g., prerequisites to fluency and comprehension, assumes concepts of print)
rationale/evidence supporting these specific intervention targets at this particular time

8. Specifying the theory of change
How the intervention addresses the need and why it should work
content: what the student should know or be able to do; why this meets the need
pedagogy: instructional techniques and methods to be used; why appropriate
delivery system: how the intervention will arrange to deliver the instruction
Most important: What aspects of the above are different from the counterfactual condition
What are the key factors or core ingredients most essential and distinctive to the intervention

9. Logic models as theory schematics
Target
Population
Intervention
Proximal Outcomes
Distal Outcomes
Positive attitudes to school
4 year old pre-K children
Exposed to intervention
Improved pre-literacy skills
Increased school readiness
Greater cognitive gains in K
Learn appropriate school behavior

11. Mapping variables onto the intervention theory: Sample characteristics
Positive attitudes to school
4 year old pre-K children
Exposed to intervention
Improved pre-literacy skills
Increased school readiness
Greater cognitive gains in K
Learn appropriate school behavior
Sample descriptors:
basic demographics
diagnostic, need/eligibility
identification
nuisance factors (for
variance control)
Potential moderators:
setting, context
personal and family
characteristics
prior experience

12. Mapping variables onto the intervention theory: Intervention characteristics
Positive attitudes to school
4 year old pre-K children
Exposed to intervention
Improved pre-literacy skills
Increased school readiness
Greater cognitive gains in K
Learn appropriate school behavior
Independent variable:
T vs. C experimental
condition
Generic fidelity:
T and C exposure to the
generic aspects of the
intervention (type,
amount, quality)
Specific fidelity:
T and C(?) exposure to
distinctive aspects of
the intervention (type,
amount, quality)
Potential moderators:
characteristics of personnel
intervention setting, context
e.g., class size

13. Mapping variables onto the intervention theory: Intervention outcomes
Positive attitudes to school
4 year old pre-K children
Exposed to intervention
Improved pre-literacy skills
Increased school readiness
Greater cognitive gains in K
Learn appropriate school behavior
Focal dependent variables:
pretests (pre-intervention)
posttests (at end of intervention)
follow-ups (lagged after end of
intervention
Other dependent variables:
construct controls– related DVs
not expected to be affected
side effects– unplanned positive
or negative outcomes
mediators– DVs on causal
pathways from intervention
to other DVs

14. Main relationships of (possible) interest
Causal relationship between IV and DVs (effects of causes); tested as T-C differences
Duration of effects post-intervention; growth trajectories
Moderator relationships; ATIs (aptitude-Tx interactions): differential T effects for different subgroups; tested as T x M interactions or T-C differences between subgroups
Mediator relationships: stepwise causal relationship with effect on one DV causing effect on another; tested via Baron & Kenny (1986), SEM type techniques.

15. Formulation of the research questions
Organized around key variables and relationships
Specific with regard to the nature of the variables and relationships
Supported with a rationale for why the question is important to answer
Connected to real-world education issues
What works, for whom, under what circumstances, how, and why?

16. Describing and Quantifying Outcomes
Mark W. Lipsey
Vanderbilt University
IES/NCER Summer Research Training Institute, 2010

17. Outcome constructs to measure
Identifying the relevant outcome constructs follows from the theory development and other considerations covered in the earlier session
What: proximal/mediating and distal outcomes
When: temporal status– baseline, immediate outcome, longer term outcomes
What else:
possible positive or negative side effects
construct control outcomes not targeted for change

18. Policy relevant outcomes
Aligning the outcome constructs and measures with the intervention and policy objectives
Instruction
Assessment
Policy relevant outcomes
(e.g., state achievement standards)

19. Alignment of instructional tasks with the assessment tasks
Identical
Instructional tasks,
activities, content
Analogous
(near transfer)
Generalized
(far transfer)

20. Basic psychometric issues
Validity (typically correlation with established measures or subgroup differences)
Reliability (typically internal consistency or test-retest correlation)
standardized measures of established validity and reliability
researcher developed measures with validity and reliability demonstrated in prior research
new measures with validity and/or reliability to be investigated in present study

21. Type of Outcome Measure
Sensitivity to change: Achievement effect sizes from 124 randomized education studies
Type of Outcome Measure
Mean Effect
Size
Number of Measures
Standardized test, broad
.04
103
Standardized test, narrow
.28
426
Focal topic test, mastery test
.40
300

22. Data from which measurement sensitivity can be inferred
Observed effects from other intervention studies using the measure
Mean effect sizes and their standard deviations from meta-analysis
Longitudinal research and descriptive research showing change over time or differences between relevant criterion groups
Archival data allowing ad hoc analysis of, e.g., change over time, differences between groups
Pilot data on change over time or group differences with the measure

23. Variance control and measurement sensitivity
Variance control via procedural consistency and statistical control using
covariates for e.g., pre-intervention individual differences and differences in testing procedures or conditions

24. Issues related to multiple outcome measures

25. Correlated measures: overlap and efficiency
Factor Analysis of Preschool Outcome Variables
Subtest
Factor Loadings
Pre-K
Pretest
Posttest
Kindergarten
Follow-up
Letter Word Identification
Quantitative Concepts
Applied Problems
Picture Vocabulary
Oral Comprehension
Story Recall
.60
.82
.75
.53
.69
.80
.76
.79
.55
.73
.78
.67
.74
.61

26. Correlated change may be even more relevant
Factor Analysis of Gain Scores for Pre-K Outcomes
Subtest
Factor Loadings
Pre to
Post
Post to
Follow-up
Basic School Skills
Letter Word Identification
Quantitative Concepts
Applied Problems
Complex Language
Picture Vocabulary
Oral Comprehension
Story Recall

27. Handling multiple correlated outcome measures
Pruning– try to avoid measures that have high conceptual overlap and are likely to have relatively large intercorrelations
Procedural– organize assessment and data collection to combine where possible for efficiency
Analytic
create composite variables to use in the analysis
use multivariate techniques like MANOVA to examine omnibus effects as context for univariate effects
use latent variable analysis, e.g., in SEM

28. IES Guidelines on multiple significance tests
Schochet, P.Z. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations. IES/NCEE
Delineate separate outcome domains in the study protocol.
Define confirmatory and exploratory analysis prior to data analysis
Specify which subgroups will be part of the confirmatory analysis and which will be part of the exploratory analysis
Design the evaluation to have sufficient statistical power for examining effects for all prespecified confirmatory analyses
For domain-specific confirmatory analysis, conduct hypothesis testing for domain outcomes as a group
Multiplicity adjustments are not required for exploratory analysis
Qualify confirmatory and exploratory analysis findings in the study report

29. Practicality and appropriateness to the circumstances
Feasibility– time and resources required
Respondent burden– minimize demands, provide incentives/compensation
Developmental appropriateness– consider not only age but performance level, possible ceiling and floor effect
For follow-up beyond one school year, may need measures designed for a broad age span to maintain comparability
May need to tailor measures or assessment procedures for special populations (disabilities, English language learners)