1. The Why and How of Using Case Studies to Evaluate Educational Programs Eric Barela Los Angeles Unified School District Sheryl Gowen Georgia State University

2. NCLB  Enacted in 2001  Unprecedented accountability for participants in federally-funded educational programs  Attempt to redefine scientifically-based research

3. Scientifically-based Research  “Research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs” (NCLB, 2001)  Experiments held up as gold standard of scientifically-based research

4. Implications of Scientifically-based Research  Hierarchy of research methods  Lack of consideration of process-oriented questions that must be answered using qualitative methods  Difficulty implementing experiments and quasi- experiments in school districts  Scientifically-based research definitions should include findings generated through qualitative methods (Berliner, 2002; Erickson & Gutierrez, 2002; Feuer, Towne, & Shavelson, 2002)

5. Case Study Rationale  “A case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident” (Yin, 2003, p. 13)  Long been considered an appropriate tool for conducting evaluations  Useful to determine the activities, issues, personal relationships that speak to program quality and effectiveness (Stake, 2004)

6. Case Study Methodology  Development of case study protocol (Yin, 2003) Questions Theoretical framework Data collection procedures Report outline  Multiple data sources (interviews, observations, documents, etc.)

7. Rigor of Design  Kidder & Judd (1986) propose three tests and Yin (2003) outlines strategies for each test 1. Construct validity  Multiple data sources  Member checking 2. External validity  Theoretical framework 3. Reliability  Case study protocol

8. LAUSD Charter Renewal Evaluations (Barela, et al., 2004)  State-mandated evaluation for charter renewal conducted by charter-granting agency early in last year of charter  Difficult to separate school context from charter school reform strategies  LAUSD looked to charter schools to provide potential districtwide reform strategies

9. Rationale  Each charter school treated as its own case 02-03: 4 03-04: 4 04-05: 10  Theoretical framework-Charter contract developed by each school  Provide feedback to each school as well as to charter oversight office

10. Method  Small team of data collectors (2-5) collect data for 2-3 month period  Team review of charter contract Purpose/mission statement Identification of educational programs Assessment tools Professional development Governance Parent and community involvement  Familiarization with specific programmatic components and goals

11. Method (continued)  Preliminary site visit Initial interview with key participants and site tour Establish rapport and clarify objectives  Gather relevant documents Lesson plans Rubrics Assessments Behavior management strategies Public relations and marketing materials  Observe relevant meetings School operations Instruction Parent and community involvement

12. Method (continued)  Observe classroom instruction Random sample stratified by important variables for that particular school Grades, schedules, English learners, special education students, etc. Multiple days  Interview key participants Administrators Teachers Parents

13. Data Analysis  Extent to which charter school was implementing its charter  When there were discrepancies, processes and rationales could be captured and were reported  Student outcome measures also reported

14. Report of Findings  Findings reported to individual schools and district charter oversight office during final year of charter  Opportunity for schools and charter oversight office to revise charter renewal petition  Real-world charter implementation provided to decision makers (Board of Education)

15. Implications  Evaluation regularly used by schools since findings reflected individual school contexts  Development of charter school promising practices study (Daley, 2005)

16. Latino Academic Achievement Case Study Project (Barela, Fernandez, & Hayes, 2005)  Board of Education resolution aiming to examine promising practices for narrowing the achievement gap between Latino and White elementary school students in LAUSD  School-level context was critical for framing phenomenon

17. Rationale  Identification of promising practices that could be developed into districtwide reforms  Theoretical framework-Literature on effective schooling practices for Latino students and resident knowledge of LAUSD

18. Theoretical Framework  Instructional and managerial leadership  Quality of instructional program and instructional activities  School climate and culture  Appropriate use of student assessment  Commitment to school improvement  Staff development  Culturally relevant and responsive education

19. Method  Identification of schools Latino gains on California Standards Test over two-year period most outpaced White student gains within school (or compared to district average when less than 50 White students) Comparison schools where Latino students lost the most ground over same two-year period  9 schools narrowing the gap and 3 schools widening the gap

20. Method (continued)  Teams of 1-2 data collectors collected data in schools for a two-month period  Two-day observations of 4 first-grade and 4 fifth-grade classrooms Natural practices Beginning and ending of elementary school experience in LAUSD

21. Method (continued)  Observation of relevant meetings Schoolwide professional development Grade-level Parent  Interviews Principal/Assistant Principal Title I, Bilingual Coordinators Observed teachers

22. Data Analysis  Case study reports written for each school site based on theoretical framework  Preliminary findings developed from cross- case analysis of reports  More extensive analysis conducted with all raw data to verify case study report preliminary findings

23. Implications  Findings presented to Board of Education, which created a special committee to examine achievement gap concerns  While not an evaluation, it served to frame decision maker discussions of the achievement gap and may shape future evaluations of district interventions

24. For Further Information http://perb.lausd.net