The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community Schools

Agenda 2 Research summary Community Schools Evaluation Toolkit Discussion Work time: work on your evaluations and network

Research Summary 3 Are community schools effective at improving outcomes for students, families, and the community? Collected 153 studies of community school and community school-like initiatives – CS models: lead-agency, community agency/CBO, university- assisted, and staff-initiated – Scale: boutique, local, state, national No studies in peer-reviewed journals

Characterizing the Literature 4 Mix of internal/external evaluators Great variation in study design – Process/implementation studies (e.g., number of students served) – Outcome studies (e.g., improved achievement) Number of outcomes depends on the unique strategies of each community school – found at least 30 outcomes – Tied to theory of action Selected 22 quasi-experimental studies (comparison group, interrupted time series, controls)

Achievement 5 The most rigorous studies indicate that community schools are improving student achievement for some students. High-implementing CIS schools scored +6% than comparison non-CIS schools in percent proficient in grade 8 math (p<.01) and +5.1% in grade 8 reading (p<.05) Mixed results out of Chicago for school-level data – Student-level demonstrates that OST participation matters More students who participated in Children’s Aid Society (CAS) after-school programs demonstrated a steady increase from 2004 to 2007 in their math performance levels as measured by the state assessment compared to students who did not attend CAS activities (p<.05)

Attendance 6 The most rigorous studies indicate that community schools are improving attendance. CIS: High-implementing elementary schools had higher attendance levels than comparison group (+0.2%, p<.05). So did high-implementing high schools (+0.3%, p<.01) CAS: students who participated in CAS after-school programs for 3-4 years had better attendance than students with lower or no participation (p<.05) SF Beacons: participants who attended 30+ days of Beacon after-school had 3.9% less total days of unexcused absences than those who participated less than 30 days (p<.001)

Graduation Rate 7 While a focus of many community schools, there is little evidence of improving the graduation rate. CIS: high-implementing community schools had a significantly better graduation rate than comparison schools (+4.8 percent, p<.01) – Compared to other large-scale dropout programs, high- implementing CIS schools had the highest effect size for graduation rate (ES=.31) and second-highest for dropout rate (ES=.36) CPS: schools that have been in CSI longer have better rates of ninth-grade students “on-track” to graduate than comparison group (p<.05)

Behavior 8 While a focus of many community schools, there is little evidence of improving student behavior. CPS schools had significantly less disciplinary incidents than their matched comparison group from 2002 to 2006 CAS teachers reported that they saw greater improvement in students getting along with others for students with higher rates of participation in CAS than comparison students during the school year (significance not reported)

Other Outcomes 9 Overall, few studies measure, evidence is weak, for the following outcomes:  Parental engagement  Relationships with adults  Student engagement  Impact on instruction

Future Directions for Evaluating Community Schools 10 Utilize high-quality study designs that establish causality  Why? Funding. Have to demonstrate effectiveness and grant-makers are increasingly aware of study design (e.g., i3) Evaluate the CS strategy, not just programs  Community schools are more than after-school programming  Is the strategy or particular programming causing changes in outcomes? Use appropriate unit of analysis (school, student, or both) Measure:  level of implementation: fidelity to design matters (e.g., CIS)  longevity  intensity of the intervention (after-school participation, receiving health services, … Evaluate outcomes for families and community; also student health Develop standard measures: e.g., how do you measure student engagement? Assess effectiveness of particular CS components to modify and strengthen the design (e.g., do CS coordinators matter?)

Community Schools Evaluation Toolkit 11 Rationale for Results Framework John W. Gardner Center Community School Evaluators Coalition for Community Schools JP Morgan Chase

What is in the Toolkit? Page step-by-step manual Coalition for Community Schools Logic Model Indicators for 5 short-term results Indicators for 4 long-term results Descriptions of school, city, county, and state data that is available to sites 45 surveys –for sites that are ready to collect additional data Examples from real sites

Goals of the Toolkit 13 Introduce the Coalition for Community Schools (CCS) Results Based Logic Model Support sites to use existing data sources and collect additional data (if needed) Support sites to identify areas of success and areas in need of improvement Provide evaluation planning tools

4 Parts to the Toolkit: Before you Start: Begin with the End in Mind 2. Get Ready: Prepare to Evaluate 3. Get Set: Designing the Evaluation 4. Go!: The Evaluation Process

15

Discussion 16 Describe your experience evaluating CS. What challenges have you experienced? Successes? What are your evaluation needs?

Work Time 17 Work on your evaluations and network

How to Get the Toolkit 18 Coalition for Community Schools Web site: Questions? Comments?

Contact information 19 Reuben Jacobson  Shital C. Shah 