1. December 2007 RMC Research Corp. 1 Washington State LASER Highlights From Recent Evaluation Studies Dave Weaver RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201

2. December 2007RMC Research Corp.2 Agenda For This Segment ► Impact on Student Achievement Studies  Student-Level analysis of 2006 Science WASL results  School-Level analysis of 2007 Science WASL results  West Valley School District ► Implementation Study  Study of Highly Successful Schools ► Evaluation activities for 2007-2008

3. December 2007 RMC Research Corp. 3 Student-Level Analysis of 2006 Science WASL Results Spring & Summer 2007

4. December 2007RMC Research Corp.4 Student-Level Analysis of Grade 5 WASL Data for 2006 ► Factors considered  Student demographics (i.e., socioeconomics, race/ethnicity)  Grade 5 science WASL results for 2006  Grade 4 math and Reading WASL results for 2005  The number of LASER PD hours the student’s Grade 5 teacher received each year and prior to the administration of the 2006 WASL  The number of LASER PD hours the student’s Grade 4 teacher received each year and prior to the administration of the 2005 WASL.

5. December 2007RMC Research Corp.5 2-Years of Teacher Influence 2006 Student- Level Science Data for Grade 5 2005 Student- Level Math Data for Grade 4 PD Hours of Student’s Grade 5 Teacher in 2006 PD Hours of Student’s Grade 4 Teacher in 2005 Linked By Student Linked By Teacher

6. December 2007RMC Research Corp.6 Grade 5 Regression Analysis Results VariableBetatp FRL–0.097–23.389<0.001 Hispanic–0.020–3.4670.001 Asian0.0336.811<0.001 White0.07011.160<0.001 2005 Math Scale Score 0.688174.346<0.001 06 & 05 Teacher PD 0.0246.496<0.001 Dependent Variable: 2006 Grade 5 Science Scale Score

7. December 2007RMC Research Corp.7 Factors Influencing 2006 Grade 5 Science WASL Results 32,228 Students 860 Schools 1325 Classes R 2 =.645

8. December 2007RMC Research Corp.8 Grade 8 Regression Analysis Results VariableBetatp FRL–0.073–9.559<0.001 White0.0648.512<0.001 2005 Math Scale Score 0.747100.039<0.001 06 & 05 Teacher PD 0.0385.369<0.001 Dependent Variable: 2006 Grade 8 Science Scale Score

9. December 2007RMC Research Corp.9 Factors Influencing 2006 Grade 8 Science WASL Results 7,432 Students R 2 =.634

10. December 2007RMC Research Corp.10 Finding 1 ► ► After controlling for demographics and entry skill levels of students, the results reveal that the current year of LASER professional development is a significant positive predictor of scores on the science WASL results. ► ► If all things remain equal, 25 hours of professional development translates into a 1-point gain in the science scale score.

11. December 2007RMC Research Corp.11 Finding 2 ► The professional development provided during the year that the teacher had the student in class had the greatest impact on student performance. ► The accumulated effect of previous years of professional development diminished predictive value.

12. December 2007RMC Research Corp.12 Finding 3 ► ► The current year of professional development of the 4 th and 5 th grade teachers is a significant positive predictor of student subscale scores for:   Inquiry in Science (.039)   Open-ended responses (.029)   Applications of Science (.020)

13. December 2007RMC Research Corp.13 Finding 4 ► Factors beyond the control of LASER (entry skills, demographics, poverty) account for 90% of knowable variance. ► Changes in any of these other factors can easily overshadow the positive effect of LASER professional development.

14. December 2007 RMC Research Corp. 14 School-Level Analysis of 2007 Science WASL Results Fall 2007

15. December 2007RMC Research Corp.15 What We Did ► Data Sources  School Level WASL Data from 2007  LASER Professional Development Database  School Demographic Data for 2006-07 ► Regression Analysis  Independent Variable: ► Total PD hours of core teachers in a school for 2 years prior to 2007 WASL per FTE  Dependent Variable: ► Percent of students in the school that met the Grade 5 science standard on the 2007 WASL  Control Variables: ► Percent of students who qualify for free or reduced price lunch ► Percent of students in the school that met the Grade 4 math standard on the 2006 WASL ► Students per classroom teacher

16. December 2007RMC Research Corp.16 Grade 5 Regression Analysis Results VariableBetatp 2006 Grade 4 Math 0.48111.294<0.001 FRL–0.400–9.425<0.001 Class size -0.088-2.7100.007 2-Year PD Per FTE 0.0983.0490.002 R 2 =.617

17. December 2007RMC Research Corp.17 Findings ► The number of hours of LASER professional development per FTE over a 2 year period prior to the WASL is a significant predictor of student achievement on the Grade 5 science WASL above and beyond what can be explained by the number of students who qualify for free or reduced price lunch, class size, or student skills. ► This finding was not evident at Grade 8.

18. December 2007RMC Research Corp.18 2-Year PD Hours Per FTE vs. Student Achievement

19. December 2007RMC Research Corp.19 Quartile Grouping of Schools PD Per FTE Rank Schools Adjusted Mean 3.75 or less 8530.56% 3.75 to 7.69 9832.73% 7.69 to 13.42 9934.50% More than 13.42 10337.01%

20. December 2007RMC Research Corp.20 Total School 2-Year PD Hours Per FTE Quartile Ranking vs. Percent of Students Who Met The Grade 5 Science WASL in 2007 Note: Means adjusted for Percent of students on free or reduced price lunch, class size, and previous year mathematics achievement.

21. December 2007RMC Research Corp.21 Distribution of Study Schools By LASER Alliance

22. December 2007 RMC Research Corp. 22 Analysis of West Valley School District Data Fall 2005

23. December 2007RMC Research Corp.23 West Valley School District study ► Peter Finch, Assistant Superintendent ► Pilot test involving phased implementation of inquiry-based instructional materials ► Noticed an important relationship between the number of modules students experienced and Grade 5 science WASL scores.

24. December 2007RMC Research Corp.24 Science WASL – WVSD Grade 5 % students meeting standard Students receiving LASER Science instruction State Average

25. December 2007RMC Research Corp.25 Regression Analysis of 2004 Grade 5 Science WASL Results VariableCoefficientSignificance White0.18<0.001 SPED-0.140.009 FRLN/A Modules0.150.003

26. December 2007RMC Research Corp.26 Regression Analysis of 2005 Grade 5 Science WASL Results VariableCoefficientSignificance White0.18<0.001 SPED-0.34<0.001 FRL-0.16<0.001 Modules0.18<0.001

27. December 2007RMC Research Corp.27 Conclusion The number of science modules that students experience has a significant positive impact on their science WASL achievement above and beyond what can be explained by demographic factors. (FRL, Special Education, & %White)

28. December 2007 RMC Research Corp. 28 Study of Highly Successful Schools Winter & Spring 2007

29. December 2007RMC Research Corp.29 Purpose of Study ► To identify schools that have demonstrated a consistent increases in the percentage of students who met the science standard on the Washington State Assessment of Student Learning (WASL). ► Interview the staff of those schools to determine to what they attribute their success.

30. December 2007RMC Research Corp.30 Criteria for Selecting of Schools ► Focus on Grade 5  Very few schools could demonstrate consistent improvement at Grade 8 and 10. ► At least 10 Grade 5 students complete the science WASL in 2004, 2005, and 2006. ► An increase of at least 6% or more students meeting the standard each year. ► An increase of at least 17.5% or more students meet the science standard across the 2 time intervals. ► Started with at least 20% of the students meeting the science standard in 2004.

31. December 2007RMC Research Corp.31 Description of Selected Schools ► 40 schools selected for interview ► 18 (45%) were in districts served by a LASER Alliance. ► 31 (78%) were in districts that participated in at least 1 LASER Strategic Planning Institute ► 10 (25%) are in districts that participate in one of the state MSP projects ► 4 (10%) are participating in the NCOSP ► 1 school did not participate in the interviews

32. December 2007RMC Research Corp.32 Interviews ► During December 2006 and January 2007. ► Conducted By:  RMC Research staff  Graduate students from the science education department of Portland State University ► Using structure interview protocol  Developed with input from LASER project leadership  Training provided ► With person most knowledgeable about success  Initial contact with Principal and sometimes other school staff

33. December 2007RMC Research Corp.33 Two Tiered Approach ► Open ended initial question  Summarized increases in WASL scores.  What is your school doing to improve science teaching and learning? To what do you attribute this success? ► Followed by questions about the contribution of specific aspects such as  Professional development  Instructional materials  Assessments  LASER services

34. December 2007RMC Research Corp.34 Finding 1– Instructional Materials ► All schools used inquiry-based instructional materials – Mostly FOSS ► 49% (19) of the schools indicated that recent changes to the science instructional materials attributed to their success ► Some customized or locally developed kits ► Heavily supplemented ► Non-LASER Alliance schools use a wider variety of supplementary materials and approaches.

35. December 2007RMC Research Corp.35 Finding 2 – WASL Preparation ► Initially, 54% (21) of the schools indicated that they engage students in some form of preparation for the WASL. ► Upon questioning, 95% (37) indicated that they engage students in some form of preparation for the WASL.

36. December 2007RMC Research Corp.36 Finding 2 – WASL Preparation ► Most use WASL release items  Prepare students before WASL administration  Develop WASL-like practice assessment during the year ► Explicit emphasis on  WASL “format”  WASL vocabulary

37. December 2007RMC Research Corp.37 Finding 3 – Professional Development ► 82% (32) of the schools indicated that professional development contributed to improving student WASL scores ► PD in Non-LASER Alliance schools dominated by kit and WASL training ► PD in LASER Alliance schools is more varies  Journaling/Notebooks  Leadership  Science Content  Inquiry  Coaching, Mentoring, & Consultant  EALRs and GLEs  More Organized School-Based PD

38. December 2007RMC Research Corp.38 Finding 4 – Science Advocate ► 77% (30) of schools interviewed identified 1 or 2 people who were advocates for science education. ► Passionate about science – Made things happen ► Roles included:  Science specialist (16)  Teacher leaders (9)  Strong teacher or team (6)  Community member (3)  Administrator (2)

39. December 2007RMC Research Corp.39 Finding 5 – EALRs & GLE ► 51% (20) of the schools attributed the success of their students to increased emphasis on and alignment of curriculum to the EALR’s and GLEs.  11 are Non-LASER Alliance schools  9 are LASER Alliance schools

40. December 2007RMC Research Corp.40 Finding 6 – Instructional Time ► 38% (15) of the schools attributed their success at least partially to increased science instructional time  8 are Non-LASER Alliance schools  7 are LASER Alliance schools

41. December 2007RMC Research Corp.41 Recommendations From Schools ► Interviewees were asked:  What 2 recommendations would you give to the staff of another school working to improve science teaching and learning that would improve student performance on the science WASL? ► Responses varied considerably

42. December 2007RMC Research Corp.42 Top 6 Recommendations ► Use released items & PCAs to develop skills needed to be successful on the WASL (18). ► Provide adequate professional development (13) on materials and WASL. ► Increase science instruction time (10). ► Engage students in hands-on, inquiry-based science activities (9). ► Provide teacher collaboration time (7). ► Align with EALRs and GLEs (6).

43. December 2007RMC Research Corp.43 Conclusions ► LASER actively promotes most of the key findings from this study  Inquiry-based instructional materials  Preparing students for the science WASL  Professional development  Science advocacy (leadership)  Alignment of curriculum to EALR’s & GLEs  Increased science instructional time

44. December 2007 RMC Research Corp. 44 Evaluation Plans for 2007-08

45. December 2007RMC Research Corp.45 Sentinel Site Visits ► Identify of approximately 30 schools in 2007–08 and 30 in 2008–09 ► Schools With Significant LASER Participation ► Two Groups of 15 schools each  Demonstrated significant positive gains  Demonstrated little, no, or negative gains ► Each site visit: 1 ½ to 2 days

46. December 2007RMC Research Corp.46 Site Visit Data Collection ► Interviews with principals and/or science teacher leader ► Interviews with teachers ► Observations of at least 3 lessons where inquiry– based instructional materials are used  Based on STAR Protocol ► Rubrics for rating level of implementation of inquiry-based science ► Analyze results for significant relationships between level of implementation to student achievement

47. December 2007RMC Research Corp.47 Assessment of Student Content Knowledge ► Obtain permission to use Bellevue School District assessments for Grades 6 & 7. ► Identify teacher participants  25 first time teachers & 25 veteran teachers ► Data collection for each student  Pre and post student scores  Student demographics  Special program participation ► SPED, FRL, ELL, Migrant, etc. ► Gift certificate compensation to teachers

48. December 2007 RMC Research Corp. 48 Questions ! Dave Weaver RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 (800) 788-1887 dweaver@rmccorp.com