1. Measuring the Effectiveness of Improvement Strategies Beyond District Improvement Plans Workshop Indianapolis – November 14, 2007 Division of Compensatory Education/Title I

2. 9/15/20152 Measuring the Effectiveness of Improvement Strategies Session Outcomes 1. Participants will understand the critical components of a well-written improvement plan. 2. Participants will be able to write clear, measurable goals focused on identified student learning needs. 3. Participants will be able to identify a variety of evidence useful in determining the effectiveness of improvement actions, including the effectiveness of implementation as well as evidence of progress toward achieving desired results.

3. 9/15/20153 Measuring the Effectiveness of Improvement Strategies Key Points  Effective improvement plans have a clear logic that link the plan’s components and guides action.  Effective implementation of planned strategies begins with clear, measurable goals.  The ongoing collection of evidence and data is needed to measure effectiveness of the actions in two areas: fidelity of implementation and impact on student learning.

4. 9/15/20154 School and District Improvement Plans Serve Multiple Purposes Improvement Plans 1. Document compliance 2. Guide improvement efforts

5. 9/15/20155 A logic model depicts the theory of change behind the action.

6. 9/15/20156 Well-planned improvement efforts that follow a clear logic model can impact student achievement Research-based reading practices Focused professional development Coaching, support and feedback Formative assessments guiding implementation Establishing a strong reading culture Scott (2007) “Reading First” changed reading instruction resulting in student achievement

7. 9/15/20157 Types of Assessments for Improvement Planning Planning (needs assessments) Formative (feedback enroute) Summative (outcome and impact) Guskey(2000)

8. 9/15/20158 Effective Implementation of Strategies Begins with Clear Goals “Would you tell me, please, which way I ought to go from here?” asked Alice. “That depends a good deal on where you want to get to,” said the cat. “I don’t much care where,” said Alice. “Then it doesn’t much matter which way you go,” said the cat. Lewis Carroll From Alice’s Adventures in Wonderland

9. 9/15/20159 Research Says… The impact of setting instructional goals on student achievement ranges from 18 to 41 percentile points. (Marzano, 2003) The power of student engagement in setting and monitoring goals is second only to active participation in impacting student learning. (Wang, Haertel,& Walberg, 1994). The existence of common goals in schools was…rare and the lack of agreed upon goals makes school unique among organizations. (Schmoker, 1999).

10. 9/15/201510 Goals are often too broad or vague to guide effective action.

11. 9/15/201511 Goals are often too broad or vague to guide effective action.

12. 9/15/201512 SMART Goals: A Critical Step in Improving Student Learning SStrategic and Specific MMeasurable AAttainable RResults-based TTime-bound O’Neill & Cozemius (2006)

13. 9/15/201513 SMART Goals Tree Diagram: Making goals focused and concrete

14. 9/15/201514 Goals focus on the “greatest area of need.”

15. 9/15/201515 Indicators are the evidence used to determine if the goal is being achieved.

16. 9/15/201516 Measures are the assessments used to gauge progress on the indicators.

17. 9/15/201517 Targets are the incremental steps toward a longer-range goal.

18. 9/15/201518 Breaking down the component parts helps to focus the goal toward action.

19. 9/15/201519 Measurable goals guide thoughtful action A well-written goal framing a research- based strategy is the foundation for action that can be measured and adjusted as the improvement team implements the plan. McKeever, Nine Lessons of Successful School Leadership Teams, (2003)

20. 9/15/201520 Clearly identify what teachers need to know and be able to do

21. 9/15/201521 Identify measures of effective implementation of new practice

22. 9/15/201522 Teacher professional development should affect student achievement Institute for Education Sciences (2007)

23. 9/15/201523 An improvement plan that is useful for guiding action… Follows a logical path Identifies critical student learning needs Has clear, measurable goals Uses research-proven practices Includes measures of changing practice And measures of impact of practice on student learning

24. 9/15/201524 Bartholomew Consolidated School Corporation Columbus, Indiana Dr. George Van Horn Director of Special Education

25. 9/15/201525 Disaggregating Student Data

26. 9/15/201526 Creating Specific, Measurable Goals

27. 9/15/201527 Measuring Effect of Practices on Student Achievement

28. 9/15/201528 Framework for Evaluating Professional Development Collect and Analyze Five Levels of Information 1. Participants' reactions 2. Participants' learning 3. Organization’s support and change 4. Participants' use of new knowledge and skills 5. Students’ learning outcomes Guskey (2002)

29. 9/15/201529 Framework for Evaluating Professional Development Collect and Analyze of Five Levels of Information 1. Participants' reactions 2. Participants' learning 3. Organization’s support and change 4. Participants' use of new knowledge and skills 5. Students’ learning outcomes Guskey (2002)

30. 9/15/201530 What guideposts will help to assess progress along the way? “Feedback is the breakfast of champions.” Blanchard (2001)

31. 9/15/201531 Types of Evidence Classroom observations or Walkthroughs Principal – Coach - Peer Teacher logs or documentation of activity Journals of reflective practice Teacher portfolios Teacher surveys Instructional materials (e.g., lessons, units) Videotapes Pre-post documentation of knowledge (K-W-L) Of increased teacher knowledge and/or change in classroom practices

32. 9/15/201532 Types of Evidence Local assessments District assessments Local rubrics or scoring guides Structured warm up activities Assessments embedded in units of study Talking with students Grades Of impact of instruction on student learning

33. 9/15/201533 Types of Evidence From published formative assessments Measures of Academic Progress (NWEA) Scholastic Reading Inventory (Scholastic) Progress Assessment Series (Pearson) Texas Assessment of Knowledge & Skills (Merit) Of impact of instruction on student learning

34. 9/15/201534 Types of Evidence Of impact of instruction on student learning Descriptive information Attendance data Discipline referral data Surveys Promotion and graduation data

35. 9/15/201535 Characteristics of Measures of Effectiveness Yield reliable information Can be used efficiently Are minimally intrusive Provide feedback that guides action Are aligned with standards Are consistent with larger, summative measures Indicate progress toward a larger goal

36. 9/15/201536 A well written goal framing a research- based strategy can be measured and adjusted as the plan is implemented.

37. 9/15/201537 Formative assessments allow for monitoring of the strategy’s impact on student learning.

38. 9/15/201538 The gathered evidence guides the effective implementation of new instructional skills.

39. 9/15/201539 Activity: Identifying measures of effectiveness to guide improvement 1. Work with a LEA colleagues or join with another team/person if you are alone. 2. Discuss the sample goal statement and identify two measures of teachers’ effective implementation of the practice and two measures effectiveness of this practice for increasing student achievement. 3. Use the “Characteristics of Measures of Effectiveness” tool to review your work.

40. 9/15/201540

41. 9/15/201541 Characteristics of Measures of Effectiveness Yield reliable information Can be used efficiently Are minimally intrusive Provide feedback that guides action Are aligned with standards Are consistent with larger, summative measures Indicate progress toward a larger goal

42. 9/15/201542 Measuring the Effectiveness of Improvement Strategies  Useful improvement plans have clear logic that drives the plan’s components and guides the actions.  Effective implementation of planned strategies begins with clear, measurable goals.  A well written goal framing a research based strategy can be measured and adjusted.  Measures of effectiveness evaluate the degree of implementation with fidelity and the impact on student learning.

43. 9/15/201543 Break-out Session A set-aside time to review and revise district improvement plans and apply knowledge gained here. Available assistance from Great Lakes East Comprehensive Center: Stacy Rush – Students with disabilities Maria Derrick-Mescua – Students learning English Doug Walker Jayne Sowers Assistance with measuring effectiveness within district improvement plans

44. 9/15/201544 References Blanchard, K. & Johnson, S. (2001). The one minute manager. London: HarperCollins. Guskey, T. R., (1999). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Guskey, T. R. (2002) Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. Institute for Education Sciences, (2007). Reviewing the evidence on how teacher professional development affects student achievement. Washington, DC: U.S. Department of Education. Retrieved 11/7/07 from http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2007033.pdf

45. 9/15/201545 References, continued McKeever, B. (2003). Nine lessons of successful school leadership teams. San Francisco, CA: WestEd. Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent Research for Education and Learning. O’Neill, J. & Cozemius, A. (1999) The power of SMART goals: Using goals to improve student learning. Bloomington, IN. Schmoker, M. (1999). Results: The key to continuous school improvement (2nd ed). Alexandria, VA: ASCD. Scott, C. (2007). Reading First: Locally appreciated, nationally troubled. Washington, DC. Center on Education Policy. Retrieved 11/6/07 from http://www.cep- dc.org/index.cfm?fuseaction=document.showDocumentByID&nodeID=1&DocumentID=228

46. 9/15/201546 References, continued Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63(3), 249– 294.