Third-Party Evaluation Studies as a Basis for Determining Program Effectiveness and Improvement Needs Center for Research and Reform in Education Johns Hopkins University Steven M. Ross, Ph.D. Professor and Evaluation Director

The Johns Hopkins and EIA Partnership JHU has an MOU with EIA to market our evaluation services to its members Dr. David Andrews, Dean of the School of Education (SOE), is on the EIA board of Directors

The Johns Hopkins and EIA Partnership The SOE believe strongly that the future success of 21 st Century education must include “nontraditional forces,” such as social entrepreneurs (TFA, New School Venture Fund, others) and the $4 billion education industry as key components in American (and international) education

The Johns Hopkins and EIA Partnership SOE is also building online courses on educational leadership, entrepreneurism, education organizations, educational policy for rising executives in member companies SOE and EIA are working to endow an institute on private sector education at SOE

The Johns Hopkins and EIA Partnership We want SOE to be the place where education companies go for research and development and instruction

The Center for Research and Reform in Education: Evaluation Services Independent studies of program implementation, products, and outcomes Literature reviews and research papers on selected topics Best Evidence Encyclopedia (BEE)

Recent and Ongoing CRRE Evaluations Parent Engagement and Partnership Program (EIA) Middlebury Interactive Program (EIA) JUMP Math in NYC Middle School Matters National Institute of School Leaders Three principal preparation programs Women’s Initiative Fellowship Program High school reform in Minnesota Social-emotional learning in Northern Ireland The Leader in Me Program in two schools Pre-K and K Early Literacy English Language Learners in Texas

Types of Evaluation Studies Simplest and Least Costly ◦ Case Study Example: Examining a middle school’s use of a new computer program for supplementing math instruction

Types of Evaluation Studies Simplest and Least Costly ◦ Survey/Interview Study Example: How 325 principals who participated in online leadership training react to the program and their application of the skills taught

Types of Evaluation Studies Simplest and Least Costly ◦ Achievement Profile Study Example: Descriptive analysis of posted state assessment scores for 25 schools before and after using a new after-school program in E/LA

Types of Evaluation Studies Medium Rigor and Cost ◦ Mixed-Methods Control Group Study Example: Program Schools A and B are compared on district science assessments to Control Schools C and D

Types of Evaluation Studies Medium Rigor and Cost ◦ Quantitative Control Group Study Example: Using statistical controls, comparisons are made on school-level AP scores in chemistry between 26 program schools and 50 control schools

Types of Evaluation Studies Medium Rigor and Cost ◦ Qualitative Control Group Study Example: Through observations, interviews, and surveys, teaching methods and student engagement are compared at two schools receiving professional development in project- based learning and two control schools

Types of Evaluation Studies Most Rigorous and Costly (Often funded by federal grants) ◦ Mixed-Methods Randomized Comparison Study Example: 10 schools randomly selected to use a new program are compared on student-level test scores and qualitative measures to 10 schools randomly selected to serve as control sites

Types of Evaluation Studies Most Rigorous and Costly (Often funded by federal grants) ◦ Mixed-Methods Matched Comparison Study Example: 10 schools that elected to use a new program are compared on student-level test scores and qualitative measures to 10 matched schools serving as control sites

What Determines Rigor? Multiple measures (triangulation) Standardized (unbiased/objective) measures Treatment-control group comparisons Equivalent comparison groups

What Determines Cost? Accessibility of data Cooperativeness of participants Travel

The Best Study for You: Major Considerations What questions do you want to answer? How quickly do you need the answers? What resources are available to fund the study? How accessible are participants and data?

Steven M. Ross Evaluation Director, CRRE