School Improvement Facilitators Network May 7,

Target #1 Program Evaluation Tool Target #2 Data Discovery Determine Focal Points Discover Facts/Hypothesis Target #3 Interpret Proficiency Targets Beginning Plan Development

Agenda 8:30-8:40 Introductions and review process 8:40-10:00 Progress Monitoring with Program Evaluation Tool 10:00-10:10 Break 10:10-11:30Data Discovery 11:30-12:00 Facts/Hypothesis 12:00-12:30 Lunch 12:30-1:00Goal Building 1:00-3:00Facilitated Work Time for Goal Building 3:00-3:30Evaluation, Feedback and Next Steps 3

Working Agreements Participate Fully Press for Clarification Collaborate Share your Thinking Calling back to whole group with hand signal 4

5 Continuous District Improvement Facilitators Network Meetings Vision/Mission of CSIF/CDIF Network – It is the mission of the Jackson County School Improvement Consortium to support a community of collaboration using a Continuous School Improvement Process in order to increase student achievement.

6 Continuous School Improvement Facilitators Network Meetings School Improvement Support September 25 – Planning for Implementation November 13 – Got SIP now what? January 22 – School Process Rubrics March 26 – School Data Profile / Analysis May 7 – School Improvement Plan

The Big Picture

MDE Program Evaluation Tool Learning Outcomes for Schools Understand the role of program evaluation within the continuous improvement process Understand the purpose, benefit, and process of program evaluation Learn about the MDE Program Evaluation tool and resources to help to evaluate a strategy/program/initiative/reform strategy

At the top of an index card, identify a hobby, sport, or activity in which you enjoy participating. Then identify the following: 1. What would you have to do to be ready to participate? 2. What knowledge and/or skills would you need? 3. What opportunities would need to be present? 4. How would you know if you were carrying out the activity in the way it was intended? 5. What would be the result if you were skilled at the activity? Making Connections

Activity: Why, How, What? Why: Why is it important to strategically implement, monitor, and evaluate the strategy/program/initiative/reform strategy? How: How will we communicate the plan to all stakeholders so that they clearly understand and own their roles in implementation? How will we ensure implementation with fidelity? How is the implementation and impact of your plan monitored and evaluated? What: What will each strategy/program/initiative/reform strategy look like when it is implemented with fidelity? What is the expected impact on student achievement?

Why is MDE Requiring Program Evaluation? To positively impact student achievement and close gaps for the subgroups To ensure that high quality planning, implementation and evaluation are part of the Continuous Improvement Process To ensure ongoing engagement of multiple stakeholders (students, teachers, parents/community, administrators) in the planning and evaluation process To maximize the use of resources to impact student learning To provide documentation of program implementation to inform future decision-making To meet state and federal requirements

State and Federal Requirements ❑ Annual evaluation of the implementation and impact of the School Improvement Plan ❑ Modification of the plan based on evaluation results ❑ Annual evaluation of all federal programs—effectiveness & impact on student achievement, including subgroups ❑ Modification of the plan based on evaluation results MICHIGANFEDERAL ISDs/RESAs are required by PA25 to provide technical assistance to schools and districts to develop annual evaluations. ESEA requires annual evaluations of programs funded by the federal programs such as Title I, Part A, C, D; Title II and Title III.

Program Evaluation Timeline District/School Improvement Plans for : Include program evaluation activities to support Program Evaluation as part of the Continuous Improvement Process Implement Program Evaluation activities throughout the school year Summer 2015 and Beyond Sustain professional learning to discuss successes, challenges, and dandy necessary follow-up training materials and support systems June 30, 2015 Program Evaluation submitted in ASSIST A completed program evaluation using the MDE Program Evaluation Tool will be required for submission of the Consolidated Application for 2015 – 2016.

What to Evaluate? Schools are required to select one: strategy/reform strategy program initiative that would have the greatest impact on student achievement and close the achievement gaps.

What to Evaluate? Districts are required to select one: strategy/reform strategy program initiative that would most impact on student achievement and close the achievement gaps.

Where is program evaluation in the Continuous Improvement Process?

Are the RIGHT PEOPLE… Doing the RIGHT THINGS… In the RIGHT WAY… At the RIGHT TIME… …for the benefit of students?

Where is program evaluation in the Continuous Improvement Process?

20 Did your plan include activities to monitor and evaluate? ➢ Monitor Implementation ➢ Evaluate Implementation Adult Focused Student Focused ➢ Monitor Impact ➢ Evaluate Impact

LEADERSHIP COMPETENCY ORGANIZATION TRAINING COACHING SELECTION PERFORMANCE ASSESSMENT TECHNICAL ADAPTIVE DATA-DRIVEN DECISION MAKING LEADERSHIP SUPPORT SYSTEM SUPPORT MDE PROGRAM EVALUATION What is/was the readiness for implementing the program/ initiative/strategy or activity? Do/did the participants have the knowledge and skills to implement the program? What is/was the program’s impact on students? Is/was the program implemented as intended? Is/was there opportunity for high quality implementation?

Planning: How will we ensure ….. ? Plan forward… Get Ready 1. Readiness? 2. Knowledge/Skills? 3. Opportunity? Implement 4. Implement With Fidelity? Monitor/ Evaluate 5. Impact on Students? Evaluation: To what extent was there/did we…..? …to evaluate impact. Here now!

23 Leadership and Learning Center 2010

24

Baseline PM 1, Formative PM 2, Formative PM 3, Formative PM 4 = Program Evaluation; Summative Progress Monitoring → Program Evaluation

Questions for Evaluation Impact on students? Knowledge and skills? Readiness? Opportunity? Implemented as intended?

The MDE Program Evaluation Tool

When to Use the MDE Evaluation Tool MDE Tool Planning Before Identify and avoid gaps, pitfalls, and potential barriers; shape program planning During Make mid-course corrections EvaluationAfter Learn why results turned out as they did

Plan forward… Get Ready 1. Readiness? 2. Knowledge/Skills? 3. Opportunity? Implement 4. Implement With Fidelity? Monitor/ Evaluate 5. Impact on Students? Planning: How will we ensure ….. ? Evaluation: To what extent was there/did we…..? …to evaluate impact. Here now!

30 AdvancEDASSISTPLATFORM

The Program Evaluation Tool has 5 Sections & a Set of Conclusions IMPACT: What is the IMPACT of the STRATEGY/ PROGRAM/ INITIATIVE ON STUDENT ACHIEVEMENT? 1. What is the READINESS for implementing the strategy/ program/initiative? 2. Do participants have the KNOWLEDGE AND SKILLS to implement the program? 3. Is there OPPORTUNITY for implementation? 4. Is the program IMPLEMENTED AS INTENDED?

32

Impact IMPACT: What was the impact of the strategy/program/initiative on student Achievement? IN AN IDEAL STRATEGY/PROGRAM/INITIATIVE, the school’s achievement results on state or district wide assessments meet proficiency standards. Achievement gaps between each of the relevant subgroups and their counterparts have been narrowed as proposed in the School Improvement Plan’s measurable objectives. Interim assessment results indicate progress toward proficiency for all students to the satisfaction of all stakeholders. a)What is the evidence and what does it show regarding achievement of the measureable objective for all students when compared to baseline state and local data? b) What is the evidence and what does it show regarding achievement of the measureable objective for subgroups and their counterparts when compared to baseline state and local data? c) What is the evidence and what does it show regarding stakeholder (staff, parents, students) satisfaction with the results?

Now What? If Objectives were met Conclusion Were objectives met? Determine if the strategy/program/ initiative should be continued or institutionalized Analyze further using the other 4 questions Yes No Conclusion Were objectives met? Determine if the strategy/program/ initiative should be continued or institutionalized Complete the conclusion section & analyze further using the other 4 questions Determine if the strategy/program/ initiative should be continued or institutionalized & use the 4 questions for further study Determine if the strategy/program/ initiative should be continued or institutionalized & use the 4 questions for further study

Now What? If Objectives were met Conclusion Were objectives met? Determine if the strategy/program/ initiative should be continued or institutionalized Analyze further using the other 4 questions Yes No Conclusion Were objectives met? Determine if the strategy/program/ initiative should be continued or institutionalized Complete the conclusion section & analyze further using the other 4 questions Determine if the strategy/program/ initiative should be continued or institutionalized & use the 4 questions for further study Determine if the strategy/program/ initiative should be continued or institutionalized & use the 4 questions for further study

If Objectives were met Conclusion: If the objectives were met, should the strategy/program/initiative be continued or institutionalized? What is the evidence and what does it say regarding whether this was the right strategy/program/initiative to meet your needs? What is the evidence and what does it say regarding whether the benefits of the strategy/program/initiative are sufficient to justify the resources it requires? What adjustments if any might increase its impact while maintaining its integrity? What is needed to maintain momentum and sustain achievement gains? How might these results inform the School Improvement Plan?

Now What? Conclusion Were objectives met? Determine if the strategy/program/ initiative should be continued or institutionalized & use the 4 questions for further study Complete the conclusion section & analyze further using the other 4 questions Yes Complete the conclusion section AND Analyze further using the other 4 questions No

If Objectives were not met Conclusion: If the objectives were met, should the strategy/program/initiative be continued or institutionalized? What is the evidence and what does it say regarding whether this was the right strategy/program/initiative to meet your needs? What is the evidence and what does it say regarding whether the benefits of the strategy/program/initiative are sufficient to justify the resources it requires? What adjustments if any might increase its impact while maintaining its integrity? What is needed to maintain momentum and sustain achievement gains? How might these results inform the School Improvement Plan?

39 1. Readiness: What was the readiness for implementing the strategy/program/initiative?

Each section begins with a description of an ideal program 1. What is the READINESS for implementing the strategy/initiative/program? IN AN IDEAL PROGRAM, stakeholders are well-prepared to implement the program. They have read and can articulate the research foundation, and regularly use the terms in conversation with each other, students, and with parents. Staff, students and parents express a high level of interest in, support for and commitment to the program. Specific concerns have been identified and solutions have been planned/ implemented. Staff is able to seamlessly integrate the program within the context of other building/district initiatives.

Each section has 3-5 sub-questions that ask for relevant evidence 1.What is the READINESS for implementing the strategy/initiative/program? a)What evidence do you have that stakeholders can articulate and believe the research behind the decision to implement the program? b)What evidence do you have that stakeholders are committed to the program with both hearts and minds? c)What evidence do you have that stakeholder (staff, parent, student) concerns about the program have been identified and addressed? d)What evidence do you have that staff are able to integrate this program with other existing initiatives?

Each section suggests possible data sources 1.What is the READINESS for implementing the strategy/initiative/ program? a)What evidence do you have that stakeholders can articulate and believe the research behind the decision to implement the program? b)What evidence do you have that stakeholders are really committed to the program with both hearts and minds?  Possible Evidence : Professional Development materials Conference/Workshop attendance Data collection plan Suggestion box ideas collected SI team agendas Focus group interviews Data analysis work Meeting agendas/minutes Books/papers about the program Staff surveys Stakeholder survey results SI Plan elements

Finally, the section asks for a self-rating and for “ action steps” the data suggest 1.What is the READINESS for implementing the strategy/initiative/ program? Stakeholders are fully prepared. Support and commitment are generally high, but some concern or work remains. Some promising signs are mixed with major gaps in knowledge or confidence. Interest and/or commitment are low so far. Next Steps: What action steps are needed to increase readiness to undertake the program?

When is the Evaluation Submitted? Required to Complete the Tool by Spring 2015

Avoid These Pitfalls Evaluating federally funded programs separately Inclusion of many strategies/unclear on strategy Selecting ‘weak’, non-robust action steps Not addressing questions 1-4 when the initiative did not meet the objective No evidence to support high self rating on scale List of assessments rather than actual data with pre- post analysis supporting progress or lack of progress Unclear, contradictory or confusing conclusions Confusion regarding subgroups

At the top of an index card, identify a hobby, sport, or activity in which you enjoy participating. Then identify the following: 1. What would you have to do to be ready to participate? 2. What knowledge and/or skills would you need? 3. What opportunity would need to be present? 4. How would you know if you were carrying out the activity in the way it was intended? 5. What would be the result if you were skilled at the activity? Making Connections (Reference to first activity)

Michigan Proficiency Targets

85% by

Where to Find Your Targets Google: Michigan School Scorecard –Scroll to Tools –Click on List of Michigan District and School Proficiency Targets for the School Year List of Michigan District and School Proficiency Targets for the School Year –FAQs

Data Discovery

Data Targets Determine or focus on the content area requiring most attention. Determine what our summative/available data strengths and weakness are. Determine our GAP areas of focus.

The Big Picture

Professional educators honor the privacy of student, staff and family information.  FERPA Compliance  I pledge to honor the privacy and confidentiality regarding data and discussions involving students, staff and other school-related issues. I understand that the data and discussions shared at this retreat are for professional school improvement purposes. I will not divulge this confidential information to any persons outside of the professional education arena.

Data Retreat® is a registered service mark of CESA 7, WI 54 Steps- 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders

Monitoring Implementation Demographics Student Outcomes Perceptions School Processes

What data to use? What data do you use and why? Formative Interim Summative Perception Demographic Process

Data Mining Points to consider: ReadingWritingMathScienceCoordinated School Health Letter/Sound Id, Rigby, DRA, Fountas & Pinnell, QRI,STAR, EXPLORE, MEAP, MEAP Access, MI Access, PLAN, ACT, MME, COMPASS, CWCA, NWEA Writing Prompt, MEAP, MEAP Access, MI Access, Writing Prompts, Grammar Quizzes, PLAN, ACT, MME, CWCA, NWEA Addition/ Subtraction, CWCA, MEAP, MEAP Access, MI Access, Quarter Exams, PLAN, ACT, MME, COMPASS, NWEA MEAP, MEAP Access, MI Access, Physical Science Assessments, PLAN, ACT, MME, NWEA Bully Prevention, Nurtured Heart, Parent Surveys, MiPhy

58 Reading-Overall Performance

59 Reading-Grade A Gender

60 Reading-Grade B Gender

61 Reading-Grade A ED

62 Reading-Grade B ED

63 Reading-Cohort Group 43.1%

Data Interaction

©2012 AdvancED65

©2012 AdvancED66

©2012 AdvancED67

©2012 AdvancED68

Facilitated Work Time

Data Retreat® is a registered service mark of CESA 7, WI 70 Data Discovery Step #1 – Choose at least 5 of the most relevant data points from state summative data in each content area at each grade level. (For example: Gender, Economically Disadvantaged, SWD, 30%>ethnicity, ELL, Gap-top/bottom) – State Assessment Data (MISchooldata.org) – Record the facts/hypothesis discovered on the chart as your groups view data

Data Tables Fact Finding & Contributing Cause for the Gap: Hypothesis & Reason for Gap: Student Goals & Measurable Objective Statement: Strategy Statements: Activities & Action Plan : Roll Out & Sustainability Monitoring and Evaluation (Checkpoints): Primary Concerns: Primary Issues: Graphic Representation – Gap Statement Gathering Data and Team Readiness 11:45-11:50 Where are we in the process?

Data Retreat® is a registered service mark of CESA 7, WI 72 Step #2-Facts – Hypothesis Chart. Hypothesis Fact Finding/Gap Statement: 1. Observations based on student data and patterns. 2. Can’t be disputed. 3. Should include numbers such as percentages. 4. STATE THE FACTS! (Avoid words like “only” or “just”) *Example 15% of the group “all students” performed at a Level 1 on the 2013 MME Math. *NOT Our textbook series and parents are just bad. Facts 1.What are we doing that might contribute to these Results? (Possible reasons / causes for data patterns observed). 2.Explanations should come from school and classroom based factors and are about practices that can be altered. 3.Explanations should NOT be regarding characteristics of individuals or UN-alterable factors. *Example We … statements Ex: 1. We as a district don’t have a well rounded understanding of the GLCEs and HSCE taught in other grades/building. 2.We do not have the opportunity to team for strategy intervention support. NOT - Ex: 1. Too many of our students are economically disadvantaged. 2. That year of data was from a low group of students. 3. Our students have a poor home life.

Data Retreat® is a registered service mark of CESA 7, WI 73 Step #2-Facts – Hypothesis Chart. Hypothesis Fact Finding/Gap Statement: 1. Observations based on student data and patterns. 2. Can’t be disputed. 3. Should include numbers such as percentages. 4. STATE THE FACTS! (Avoid words like “only” or “just”) *Example 15% of the group “all students” performed at a Level 1 on the 2013 MME Math. *NOT Our textbook series and parents are just bad. Facts 1.What are we doing that might contribute to these Results? (Possible reasons / causes for data patterns observed). 2.Explanations should come from school and classroom based factors and are about practices that can be altered. 3.Explanations should NOT be regarding characteristics of individuals or UN-alterable factors. *Example We … statements Ex: 1. We as a district don’t have a well rounded understanding of the GLCEs and HSCE taught in other grades/building. 2.We do not have the opportunity to team for strategy intervention support. NOT - Ex: 1. Too many of our students are economically disadvantaged. 2. That year of data was from a low group of students. 3. Our students have a poor home life.

Data Retreat® is a registered service mark of CESA 7, WI 74 Data Discovery Step #3: -Narrow your focus to the top student concerns that emerged from your fact finding/observations/discussions -Determine the concerns that will serve as the basis for GOAL building for your next year School Improvement Plan.

Lunch

Goal Statement Set Measurable Objectives Building Goals in ASSIST Strategy Statement ActivitiesActivities

What is a goal vs. objective vs. strategy vs. activity? What is it?What is it’s purposeWhat does it look like? Goal StatementData-based, general statement linked to student learning needs—either Academic or Organizational “All students will be proficient in _______.” Measurable ObjectiveSpecific target that is linked to the goal. Should be “S.M.A.R.T.” “What proportion of who will do what by when as measured by ___.” StrategySpecific, research-based practice that guides educator to observable actions “Teachers will... _______” ActivitiesSpecific observable actions that a student is guided to perform. “Students will... ______”

Academic Types of Goals Organizational All students will (insert content-focused goal here) All students will (insert content-focused goal here) All students will be proficient in mathematics Increase or improve the (insert goal here) To increase the number of partnerships the school district has with businesses in the community

Setting Goals 3- 5 Goals NOT Title 1 Schools Must address any content area where school does not meet proficiency targets Title 1 Schools Must address areas identified on CNA Single goal may address more than one content area

Your turn... In the Goal Planner template, write a Student Goal Statement at the top. Each goal should be: 1.Student focused 2.Stated broadly 3.Based need identified by data analysis

SMART Measurable Objectives S pecific M easurable A chievable R ealistic T ime-bound S M A R T

Setting Measurable Objectives What proportion? Who ? What?When? Measured by? 87% Limited English Proficient 1 st and 2 nd graders demonstrate a proficiency in problem solving in math 3/14/2013 measured by the MEAP.

Design of Measurable Objective Proportion? (%) Who? (subgroup or all) What? (content & skill/strand) Measured By? (assessment/season/year) By When? (date) 100% of all students will demonstrate a proficiency in number fluency in math by moving from 46 % to 49% as measured by the State Summative Assessment by June, 2015.

State Summative—MEAP, MME, alt. Local Assessment data Norm Referenced— ACT, Explore, Plan, etc. Data Measurement Triangulation

What data is in your Measurable Objective? Plan to develop strategies around those targets Plan to monitor that data in coming school year Plan to bring current SY data to the June, 2014 Retreat

Your turn... In the Goal Planner template, record any measurable objectives for each goal Each M.O. includes 1.Proportion? (%) 2.Who? (subgroup or all) 3.What? (content & skill/strand) 4.Measured By? (assessment/season/year) 5.By When? (date)

The Big Picture

Selecting Best Practices Is this the right thing to do? Can we do it the right way?

Right Thing? Is this the right thing to do? Does the practice align to our need? Will the practice be a good fit with other district and school needs and priorities? Is This the Right Thing to Do?

Right Way? Can we do it the right way? Can we implement with fidelity? Is this the right thing to do? Methodology Frequency Fidelity Non Negotiable

Right Way? Can we do it the right way? Similar to research situation? Build commitment and readiness? Develop technical and adaptive skills in leadership? Put the right people in the right place? Build the necessary capacity, knowledge, and skills and provide on-going support? Put the required structures in place? Commit the necessary financial resources? Monitor fidelity?

How Do We Decide? Is this the right thing to do? Choose a different best practice Can we do it the right way? Proceed to Plan stage and develop an action plan Choose a different best practice Yes No

Strategy What staff members will do instructionally to help students achieve the measureable objective. This also includes the research & MTSS Tier I, II, or III identification.

Strategy-Research What Works Clearinghouse Data 4 SS Mel.org Michigan Department of Education Research Works Ask your Regional SI Coordinator for additional supporting research.

Data Retreat® is a registered service mark of CESA 7, WI 95 Next Steps-JUNE SESSION 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders

Data Retreat® is a registered service mark of CESA 7, WI 96 Who/What Who are you bringing in June? Why are you bringing them? What additional data are you bringing in June? Who are you bringing in June? Why are you bringing them? What additional data are you bringing in June?

Data Mining Points to consider: ReadingWritingMathScienceCoordinated School Health Letter/Sound Id, Rigby, DRA, Fountas & Pinnell, QRI,STAR, EXPLORE, MEAP, MEAP Access, MI Access, PLAN, ACT, MME, COMPASS, CWCA, NWEA Writing Prompt, MEAP, MEAP Access, MI Access, Writing Prompts, Grammar Quizzes, PLAN, ACT, MME, CWCA, NWEA Addition/ Subtraction, CWCA, MEAP, MEAP Access, MI Access, Quarter Exams, PLAN, ACT, MME, COMPASS, NWEA MEAP, MEAP Access, MI Access, Physical Science Assessments, PLAN, ACT, MME, NWEA Bully Prevention, Nurtured Heart, Parent Surveys, MiPhy

Data Retreat® is a registered service mark of CESA 7, WI 98 Next Steps-SEPTEMBER SESSION 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders 1.Data Discovery 2.Facts/Hypotheses 3.Proficiency targets 4.Plan Development 5.Put into ASSIST 6.Progress monitoring plan / evaluation 7.Rollout to stakeholders

Facilitated Work Time for Goal Building

Questions

Questions/Comments? Please contact:  Susan Townsend  Maeghan McCormick  Sandy White  Steve Doerr  Jennifer Fox Or visit the MDE - School Improvement websiteMDE - School Improvement