Curriculum Renewal: Fidelity of Implementation WERA/OSPI State Assessment Conference— Seattle Airport Hilton December 4, 2008 Peter Hendrickson, Ph.D. Everett Public Schools

Program Evaluation  Fidelity of Implementation Program Model Theory of Action or Logic Model To what extent, consistent?  Program Impacts Knowledge Skills Dispositions

Theory of Action  What are you doing?  Who are you doing it with?  What resources do you have?  What effects do you expect? (Green, 2008)

Logic Model fragment InputsMethodsOutputsShort Term Outcomes Medium Term Outcomes Long Term Outcomes Trainer Facilitator Lead tchr Principals Overview Intense Folo Up Drop in Study Grp All attend 85% attend 50% use Tchr uses new method K-2 form ltrs new method Score classmate reliably Students apply in school settings Grd 4, 5 Alignment New texts Tchr guide Post docs Trade old Half day FAQ All obs in 1 qtr Help accessed Note legible in parent conf Use rubric to self assess Apply in leisure settings MS, HS

FOI Critical Components  Structural/ procedural  Structural/ educative  Instructional/ pedagogical  Student engagement  What to do  Need to know  Instructional strategies  Expectations for students (Century et.al., 2007)

Degree implemented  On Model, Off Model  Various Models, e.g. Read minute Local 55 minute HS Local 45 minute MS Local 45 minute, plus 30 SSR

Direct Observation--Tools  Observation protocol  Train observers  Timely feedback  Additional information  Use the data

FOI Framework, p.1 ComponentNot PresentPartialAdequateSubstantial Fidelity What to do Procedures Physical Organize Needs to Know Content Pedagogy

FOI Framework, p. 2 ComponentNot PresentPartialAdequateSubstantial Fidelity Instruct Strategies Prior Learn Direct Instr Guided Prac Assessment Engage- ment Instruction- al Student

Zoomerang survey  Objectives  Design  Format, pilot  Administer, remind  Organize data  Analyze  Report

Focus groups  Representative  Possible time  Field test  Conduct  Transcribe, validate  Content analysis

Interviews  Go to their turf  Probe for key elements  Keep short  Listen for unexpected  Don’t argue  Act on findings

Software monitoring  Who enrolled  Time on software  Books read  Vocabulary growth  Comprehension growth  Coasting?

Student Information System  Present  Tardy  Gender  Ethnicity  Prior learning  Special programs

References  Bamberger, M., J. Rugh, & L. Mabry. (2006). Real World Evaluation. Thousand Oaks, CA: Sage Publications. Three veteran program evaluators reveal tricks of the trade with limited resources, time or commitment.  Borman, G., Slavin, R.E., Cheung, A., Chamberlain, A., Madden, N.A., & Chambers, B. (2007). Final reading outcomes of the national randomized field trial of Success for All. American Educational Research Journal, 44 (3), When implementing Success for All, considerable emphasis is placed on fidelity. External evaluators, extensive training for building principals, building-wide commitment and exhaustive instructional rubrics set the program apart from most other literacy models.Final reading outcomes of the national randomized field trial of Success for All  CEMSE. (2007). Fidelity of Implementation Questionnaires for Mathematics Curriculum in Grades 1-5., Chicago, IL: University of Chicago.  CEMSE. (2008). Critical Components: Definitions and Explanations. Retrieved November 21, 2008 from the Worldwide Web at summary 2008_11_110.pdf This fully fleshed math and science FOI checklist provides a useful base document for creating local checklists. summary 2008_11_110.pdf

References  Century, J., Freeman, C., Rudnick, M., & Leslie, D. (2007). A conceptual framework for fidelity of implementation of instructional materials. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY. March 28th  Fullan, M. (2001). Leading in a Culture of Change. San Francisco, CA: Jossey-Bass.  Hendrickson, P. (2008). Unpublished principal interview protocols for several program evaluations. Everett, WA: Everett Public Schools.  --. (2006). Read 180 Placement, Assessment and Reporting Guide. NYC, NY: Scholastic. The PARG is a companion tool to the Research Protocol and Tools (2007).  Kerman, Sam. (1979). Teacher Expectations and Student Achievement. Phi Delta Kappan, v.60, n.10, p , June  National Sciences Resource Center. (1977). Science for All Children. Washington, D.C.: National Academy Press.  Zoomerang is available free via on the Worldwide Web to implement surveys for limited use. A full license is less than $500 per year. Survey Monkey works, too.

Contact Information Peter Hendrickson, Ph.D. Curriculum Specialist Assessment, Research, Program Evaluation Everett Public Schools Tel: