Robert Barnoski (Barney) Washington State Institute for Public Policy Phone: (360) Institute Publications: Promoting Academic Success Evaluation Progress Washington Education Research Association December 7, 2006

Page 2 of 20 Washington State Institute for Public Policy Created in 1983 by the state Legislature Mission: Carry out non-partisan research on projects assigned either by the legislature or the Institute’s Board of Directors 8 legislators 4 higher education provosts or presidents 4 state agency directors

Page 3 of Legislature directed Institute to Evaluate Promoting Academic Success The evaluation should focus on determining: (a) overall effectiveness in helping students pass the WASL; (b) relative effectiveness of different remedial strategies offered; and (c) relative effectiveness by student characteristics, including economic status, limited English proficiency, and ethnicity. The office of the superintendent of public instruction shall provide all data necessary to conduct such analyses, and shall help coordinate data collection directly from districts administering the remedial programs as necessary. ESSB 6386, §607 (11), Chapter 372, Laws of 2006, supplemental operating budget.

Page 4 of 20 An interim report is due to the Legislature by December 15, The final report is due December 15, 2007.

Page 5 of 20 Analyses of 10 th Grade Spring 2006 WASL Data Reports on Website

Page 6 of 20 (a) Evaluating Overall Effectiveness of Summer 2006 PAS Compare WASL Retake Results for: Students who participated in Summer 2006 PAS and retook WASL in August, With similar group of students not in PAS but who retook WASL in August. Key is finding a similar comparison group. Statistical control variables (covariates) are still used in multivariate analysis to estimate impact of PAS participation on WASL retake. Two dependent variables: –Met standard and –Gain on WASL scale scores.

Page 7 of 20 (a) Evaluating Overall Effectiveness of Summer 2006 PAS PAS Student Data Collection: OSPI built an application for recording students participating in PAS. Concerns about privacy kept OSPI from requesting the names of the teachers who provided the PAS instruction to the PAS students. As a result, Institute must rely upon the using district, school, content area, and PAS program name to link teacher surveys to student WASL results. Institute must reply upon school districts recording which students took summer PAS. OSPI called districts to encourage them to enter Summer 2006 PAS student data. School districts have until June 2007 to enter PAS students.

Page 8 of 20 (b) Evaluating Relative Effectiveness of Different Remedial Strategies for Summer 2006 PAS Measuring Different Remedial Strategies: Institute developed three surveys for PAS teachers after discussions with OSPI staff (Reading, Writing and Math). Two versions were created: paper and pencil, and online. The teachers were to be surveyed at the end of the summer 2006 effort. We will analyze the associations between different strategies and gains on WASL retakes for students who experienced the strategies. Statistical control variables (covariates) are still used in multivariate analysis when estimating impact of PAS strategies on the WASL retake.

Page 9 of 20 (b) Evaluating Relative Effectiveness of Different Remedial Strategies for Summer 2006 PAS Remedial Strategy Data Collection: OSPI built an application to record PAS program and teacher contact information. Institute must reply upon district PAS coordinator recording the summer 2006 PAS program/teacher contact information. Not all districts entered these data before the end of summer school. School districts continued to enter Summer PAS data in August and September. Institute sent out teacher surveys as soon as we received the PAS teacher’s contact information. Institute hired WSU Social & Economic Sciences Research Center to do follow-up phone calling.

Page 10 of 20 Examples of Strategies on Math Survey for Summer 2006 PAS What assessments were used to determine student math needs at the beginning of this instruction? (Check all that apply)  Diagnostic test(s)  Classroom assignments  WASL math strand results  Quizzes/tests  Homework assignments  Teacher observation  Standardized achievement tests  WASL math scores  None  Other (specify):

Page 11 of 20 PAS teacher survey response rate is approximately 65 percent.

Page 12 of 20 In your section(s), approximately what percentage of time did you spend in each of the following areas? (Must add to 100%) __ Academic/vocational/career counseling __ General instruction for math __ Motivation __ Specific instruction on the EALRs and GLEs __ Study habits __ WASL test taking skills: guessing and scoring rules __ Other (specify): Examples of Strategies on Math Survey for Summer 2006 PAS

Page 13 of 20 Examples of Strategies on Math Survey for Summer 2006 PAS What instructional materials did you use in your section(s)? (Check all that apply)  OSPI Instructional Support Modules  WASL practice tests/released items other than from OSPI Instructional Support Modules  WASL teaching materials other than OSPI Instructional Support Modules  Materials aligned with EALRs/GLEs other than OSPI Instructional Support Modules  Materials specifically obtained for this remedial instruction other than OSPI Instructional Support Modules  District-adopted instructional materials  Technology-based materials. Name(s):  Other materials (describe):

Page 14 of 20 Examples of Strategies on Math Survey for Summer 2006 PAS What special training have you received to teach students who have not met the 10th-grade math standards? (Check all that apply)  OSPI Instructional Support Module training from OSPI staff  OSPI Instructional Support Module training from district or ESD staff  None  Other training (describe): In which of the following WASL math activities have you participated? (Check all that apply)  Item writing  Scoring  Range finding  Data review  OSPI/ESD deep alignment  Curriculum alignment  None  Other (specify):

Page 15 of 20 (c) Estimating the Influence of Student Characteristics for Summer 2006 PAS Analyze association between different strategies and gains on WASL retakes by student characteristics.

Page 16 of 20 Number of PAS Teacher Surveys Summer 2006 Subject Area Surveys Sent Surveys Received Percent Received Math % Reading % Writing %

Page 17 of 20 Math Teacher Survey Academic/vocational/career counseling 1.9% General instruction for math46.7% Motivation9.2% Specific instruction on the EALRs and GLEs17.9% Study habits4.7% WASL test taking skills: test taking, guessing, and scoring rules 17.3% Other (specify):2.3% Total100.0% In your section(s), approximately what percentage of time did you spend in each of the following areas?

Page 18 of 20 Number of PAS Student Records Summer 2006 Subject Area Records Received Math4,530 Reading1,012 Writing1,142 Total6,684

Page 19 of 20 Number of WASL Retakes Summer 2006 PAS Reading Spring 2006 Reading Level Total Below StdRetake % Retake NoL12, % NoL26, % NoTotal9, % YesL % YesL % YesAll %

Page 20 of 20 Number of WASL Retakes Summer 2006 PAS Writing Spring 2006 Writing Level Total Below Std Retake Total % Retake NoL % NoL % NoTotal % YesL % YesL % YesTotal %

Page 21 of 20 Number of WASL Retakes Summer 2006 PAS Math Spring 2006 Math Level All Retake Total % Retake NoL % NoL % NoTotal % YesL % YesL % YesTotal %

Page 22 of 20 Thank You!