Measuring the Effectiveness of Improvement Strategies Beyond District Improvement Plans Workshop Indianapolis – November 14, 2007 Division of Compensatory Education/Title I

9/15/20152 Measuring the Effectiveness of Improvement Strategies Session Outcomes 1. Participants will understand the critical components of a well-written improvement plan. 2. Participants will be able to write clear, measurable goals focused on identified student learning needs. 3. Participants will be able to identify a variety of evidence useful in determining the effectiveness of improvement actions, including the effectiveness of implementation as well as evidence of progress toward achieving desired results.

9/15/20153 Measuring the Effectiveness of Improvement Strategies Key Points  Effective improvement plans have a clear logic that link the plan’s components and guides action.  Effective implementation of planned strategies begins with clear, measurable goals.  The ongoing collection of evidence and data is needed to measure effectiveness of the actions in two areas: fidelity of implementation and impact on student learning.

9/15/20154 School and District Improvement Plans Serve Multiple Purposes Improvement Plans 1. Document compliance 2. Guide improvement efforts

9/15/20155 A logic model depicts the theory of change behind the action.

9/15/20156 Well-planned improvement efforts that follow a clear logic model can impact student achievement Research-based reading practices Focused professional development Coaching, support and feedback Formative assessments guiding implementation Establishing a strong reading culture Scott (2007) “Reading First” changed reading instruction resulting in student achievement

9/15/20157 Types of Assessments for Improvement Planning Planning (needs assessments) Formative (feedback enroute) Summative (outcome and impact) Guskey(2000)

9/15/20158 Effective Implementation of Strategies Begins with Clear Goals “Would you tell me, please, which way I ought to go from here?” asked Alice. “That depends a good deal on where you want to get to,” said the cat. “I don’t much care where,” said Alice. “Then it doesn’t much matter which way you go,” said the cat. Lewis Carroll From Alice’s Adventures in Wonderland

9/15/20159 Research Says… The impact of setting instructional goals on student achievement ranges from 18 to 41 percentile points. (Marzano, 2003) The power of student engagement in setting and monitoring goals is second only to active participation in impacting student learning. (Wang, Haertel,& Walberg, 1994). The existence of common goals in schools was…rare and the lack of agreed upon goals makes school unique among organizations. (Schmoker, 1999).

9/15/ Goals are often too broad or vague to guide effective action.

9/15/ Goals are often too broad or vague to guide effective action.

9/15/ SMART Goals: A Critical Step in Improving Student Learning SStrategic and Specific MMeasurable AAttainable RResults-based TTime-bound O’Neill & Cozemius (2006)

9/15/ SMART Goals Tree Diagram: Making goals focused and concrete

9/15/ Goals focus on the “greatest area of need.”

9/15/ Indicators are the evidence used to determine if the goal is being achieved.

9/15/ Measures are the assessments used to gauge progress on the indicators.

9/15/ Targets are the incremental steps toward a longer-range goal.

9/15/ Breaking down the component parts helps to focus the goal toward action.

9/15/ Measurable goals guide thoughtful action A well-written goal framing a research- based strategy is the foundation for action that can be measured and adjusted as the improvement team implements the plan. McKeever, Nine Lessons of Successful School Leadership Teams, (2003)

9/15/ Clearly identify what teachers need to know and be able to do

9/15/ Identify measures of effective implementation of new practice

9/15/ Teacher professional development should affect student achievement Institute for Education Sciences (2007)

9/15/ An improvement plan that is useful for guiding action… Follows a logical path Identifies critical student learning needs Has clear, measurable goals Uses research-proven practices Includes measures of changing practice And measures of impact of practice on student learning

9/15/ Bartholomew Consolidated School Corporation Columbus, Indiana Dr. George Van Horn Director of Special Education

9/15/ Disaggregating Student Data

9/15/ Creating Specific, Measurable Goals

9/15/ Measuring Effect of Practices on Student Achievement

9/15/ Framework for Evaluating Professional Development Collect and Analyze Five Levels of Information 1. Participants' reactions 2. Participants' learning 3. Organization’s support and change 4. Participants' use of new knowledge and skills 5. Students’ learning outcomes Guskey (2002)

9/15/ Framework for Evaluating Professional Development Collect and Analyze of Five Levels of Information 1. Participants' reactions 2. Participants' learning 3. Organization’s support and change 4. Participants' use of new knowledge and skills 5. Students’ learning outcomes Guskey (2002)

9/15/ What guideposts will help to assess progress along the way? “Feedback is the breakfast of champions.” Blanchard (2001)

9/15/ Types of Evidence Classroom observations or Walkthroughs Principal – Coach - Peer Teacher logs or documentation of activity Journals of reflective practice Teacher portfolios Teacher surveys Instructional materials (e.g., lessons, units) Videotapes Pre-post documentation of knowledge (K-W-L) Of increased teacher knowledge and/or change in classroom practices

9/15/ Types of Evidence Local assessments District assessments Local rubrics or scoring guides Structured warm up activities Assessments embedded in units of study Talking with students Grades Of impact of instruction on student learning

9/15/ Types of Evidence From published formative assessments Measures of Academic Progress (NWEA) Scholastic Reading Inventory (Scholastic) Progress Assessment Series (Pearson) Texas Assessment of Knowledge & Skills (Merit) Of impact of instruction on student learning

9/15/ Types of Evidence Of impact of instruction on student learning Descriptive information Attendance data Discipline referral data Surveys Promotion and graduation data

9/15/ Characteristics of Measures of Effectiveness Yield reliable information Can be used efficiently Are minimally intrusive Provide feedback that guides action Are aligned with standards Are consistent with larger, summative measures Indicate progress toward a larger goal

9/15/ A well written goal framing a research- based strategy can be measured and adjusted as the plan is implemented.

9/15/ Formative assessments allow for monitoring of the strategy’s impact on student learning.

9/15/ The gathered evidence guides the effective implementation of new instructional skills.

9/15/ Activity: Identifying measures of effectiveness to guide improvement 1. Work with a LEA colleagues or join with another team/person if you are alone. 2. Discuss the sample goal statement and identify two measures of teachers’ effective implementation of the practice and two measures effectiveness of this practice for increasing student achievement. 3. Use the “Characteristics of Measures of Effectiveness” tool to review your work.

9/15/201540

9/15/ Characteristics of Measures of Effectiveness Yield reliable information Can be used efficiently Are minimally intrusive Provide feedback that guides action Are aligned with standards Are consistent with larger, summative measures Indicate progress toward a larger goal

9/15/ Measuring the Effectiveness of Improvement Strategies  Useful improvement plans have clear logic that drives the plan’s components and guides the actions.  Effective implementation of planned strategies begins with clear, measurable goals.  A well written goal framing a research based strategy can be measured and adjusted.  Measures of effectiveness evaluate the degree of implementation with fidelity and the impact on student learning.

9/15/ Break-out Session A set-aside time to review and revise district improvement plans and apply knowledge gained here. Available assistance from Great Lakes East Comprehensive Center: Stacy Rush – Students with disabilities Maria Derrick-Mescua – Students learning English Doug Walker Jayne Sowers Assistance with measuring effectiveness within district improvement plans

9/15/ References Blanchard, K. & Johnson, S. (2001). The one minute manager. London: HarperCollins. Guskey, T. R., (1999). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Guskey, T. R. (2002) Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. Institute for Education Sciences, (2007). Reviewing the evidence on how teacher professional development affects student achievement. Washington, DC: U.S. Department of Education. Retrieved 11/7/07 from

9/15/ References, continued McKeever, B. (2003). Nine lessons of successful school leadership teams. San Francisco, CA: WestEd. Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent Research for Education and Learning. O’Neill, J. & Cozemius, A. (1999) The power of SMART goals: Using goals to improve student learning. Bloomington, IN. Schmoker, M. (1999). Results: The key to continuous school improvement (2nd ed). Alexandria, VA: ASCD. Scott, C. (2007). Reading First: Locally appreciated, nationally troubled. Washington, DC. Center on Education Policy. Retrieved 11/6/07 from dc.org/index.cfm?fuseaction=document.showDocumentByID&nodeID=1&DocumentID=228

9/15/ References, continued Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63(3), 249– 294.