Rural Dropout Prevention Project Tools for measuring the use of Early Warning data October 6, 2015 March 2015

About the Rural Dropout Prevention Project Goal: Provide technical assistance as determined through collaboration with the state education agency (SEA) to states, districts, and schools, particularly those in rural communities, to support dropout prevention efforts.

Participating States Fifteen states selected to participate (MT, WY, ND, SD, NE, IA, AR, OK, NH, VT, WV, NC, ME, AK, MS)

Projects Priority Areas Reengaging out-of-school youth Assigning adult advocates to at- risk students Implementing early-warning indicator systems Implementing effective credit recovery and acceleration opportunities Instituting “summer bridge” transition programs Improving services and instruction for English language learners (ELL) and new immigrants Improving services and instruction for migrant, American Indian, Alaska Native, minority, and at-risk students and youth Supporting in-service leadership and professional development programs Addressing other topics suggested by the SEAs that relate to or enhance the other priority areas

Gathering Input to Drive TA Planning Asked for input about: Unique challenges faced by rural communities What dropout prevention looks like in rural areas and what approaches are successful Types of support that would be helpful

Examples of What We Heard: Approaches to Dropout Prevention Early Warning Tool available from the state using predictive analytics to identify at risk students. Support whole child needs Partnering with communities and businesses Utilizing flexibility provided through innovation zone grants Focusing on school climate and student engagement Conducting site visits to colleges starting in middle school Providing alternative pathways through career and technical education (CTE) opportunities and connecting CTE and core curriculum

Examples of What We Heard: Challenges Using data to inform practice (not collecting data for data’s sake) Grant writing Changing community contexts

Focus Based on What We Heard To increase the use of data to inform dropout prevention activities in West Virginia

Early Warning Implementation Pathway Synthesize Research Validate Indicators West Virginia's Early Warning Tool Customize and Develop Tools and Supports Launch and Implement West Virginia’s Early Warning Tool uses predictive analytics to identify students at risk. Justin will be providing a short update about this next. Assess and Improve Processes

Collecting Data is Great but Using it is Critical Synthesize Research Validate Indicators Customize and Develop Tools and Supports Launch and Implement West Virginia’s Early Warning Tool uses predictive analytics to identify students at risk. Justin will be providing a short update about this next. Assess and Improve Processes

Early Warning Intervention Monitoring System Implementation Cycle STEP 1 Establish roles and responsibilities STEP 2 Use the EW tool STEP 3 Review the EW data STEP 4 Interpret the EW data STEP 5 Assign and provide interventions STEP 6 Monitor students STEP 7 Evaluate and refine the process Developed by the National High School Center in Collaboration with For more information on this process visit: http://www.earlywarningsystems.org/resources-tools/implementation-and-research

Tools to Measure Early Warning Implementation: Rubric and Interview Rubric and worksheet

Sections of the Rubric and Worksheet System Features to Support Readiness and Implementation Data Team and Structure EWS Tool Capabilities Review of EWS Data Interventions and Supports Progress Monitoring Continuous Improvement Aligned with the implementation cycle. Each of these sections include subsections. System Features to Support Readiness and Implementation. Resources and organizational structures necessary to support, identify, and intervene to support students at risk of dropping out of high school Data Team and Structure. Team structure and process in place to review EWS data and other data in order to assign and monitor interventions EWS Tool Capabilities. Components and capabilities of the tool (e.g., indicators and thresholds, reports) and maintenance of the tool to support identification of students at risk of dropping out Review of EWS Data. Data review process to identify at-risk students, groups of students, or schoolwide patterns, and to explore underlying root causes of risk and develop a hypothesis for intervention and support Interventions and Supports. Interventions and supports of varying focus (e.g., attendance, course performance, behavior) and intensity—aligned to address identified root causes of risk—provided to support dropout prevention efforts Progress Monitoring. Ongoing and frequent monitoring of student progress across levels of interventions to monitor responsiveness to interventions and supports, and to make adjustments and adaptations as needed Continuous Improvement. System for collecting and analyzing data to measure fidelity and effectiveness of EWS implementation

Implementation Rubric Includes Rating scale 1–5 points with anchors for 1 point = little or no implementation 3 points = partial or inconsistent implementation 5 points = complete and consistent implementation

Implementation Interview Goal is to create dialogue Script and note-taking template for gathering information. Aligned with rubric and includes sample questions The Implementation Interview is intended to create dialogue about implementation Do not need to use all the prompts available and can streamline based on context of school. If you are a reviewer may consider if any of the language needs to change based on school context.

Goal of the Tools The goal is not to be evaluative but to inform identification of implementation strengths, detect barriers that teams have encountered, and identify areas in need of additional focus or professional development.

Use of Tools: Administration Self-assessment: School based team works through questions and rate themselves based on the descriptions in the rubric May be beneficial for team members to individually rate themselves, compare results and discuss any differences External review: Independent reviewers hold discussion with school-based team using Implementation Interview and then independently score using the rubric Recommended that two facilitators are used to allow for a check of interrater reliability.

Use of Tools: Scoring Scored against specific language/definitions listed in the rubric When in doubt, round scores down Used scores of “2” and “4” when implementation did not exactly match definitions of a “1,” “3,” or “5.” Scores averaged across implementation area Components not in place scored a “1.” Considered evidence when determining scores

Use of Tools: Time & Frequency Estimated to take about 2hrs to conduct interview with school sites, but will depend on the depth of conversations with sites. If time limited, could target specific sections of the rubric and interview. Frequency: Collect overtime allows you to see change in implementation May consider collecting annually

Status of Tools Currently being customized to fit language for West Virginia Look for additional information about access coming soon!

Questions

Video: Sneak Peak

Thank you! Amy Peterson ampeterson@air.org