1. Virginia Middle School Research Alliance (VMSRA) Using the early warning system process to identify and meet middle schoolers needs Aimee Evan VMSRA Lead, REL Appalachia Virginia School-University Partnership Summer Leadership Conference Charlottesville, VA June 25, 2014

2. Introductions Name Role Division Previous involvement with a Regional Educational Laboratory (REL) What brought you to this session 2

3. Background on REL Appalachia

4. Regional Educational Laboratory (REL) Program U.S. Department of Education, Institute of Education Sciences. RELs provide regional support for: – Applied research and evaluation. – Technical support and information sharing to build capacity to use data for improved education outcomes. 4

5. REL Appalachia’s Mission Meet the applied research and technical support needs of Kentucky, Tennessee, Virginia, and West Virginia. Bring evidence-based information to policymakers and practitioners: – Provide support for a more evidence-reliant education system. – Inform policy and practice for states, districts, schools, and other stakeholders. – Focus on high-priority, discrete issues and build a body of knowledge over time. 5

6. What Does REL Appalachia Do? 6 Assess regional research needs by monitoring emerging education issues and challenges. Maintain and refine research alliances through ongoing dialogue between educators in each region and researchers. Provide analytic technical support to increase use of data and analysis to understand policies and programs, make decisions, and support effective practice. Conduct research and evaluation studies of rigor and method appropriate to the questions the studies attempt to answer. Distribute results of REL research across the region. Coordinate and partner with other RELs and federal, state, and local education research and technical assistance organizations.

7. Research Alliances What is a research alliance? – A partnership between education stakeholders and REL Appalachia. What is the purpose of a research alliance? – As partners, REL Appalachia and alliance members develop and carry out a research and analytic technical assistance agenda on priority topics. Who are the education stakeholders in an alliance? – May include representatives from one or more schools, divisions, state education agencies, and other organizations (e.g., colleges and universities). 7

8. VMSRA Goals

9. Superintendents, assistant superintendents, directors of curriculum and instruction, directors of assessment and testing, principals, teachers from the following divisions: Executive directors of SURN (School-University Research Network) and VSUP (Virginia School-University Partnership). Virginia Department of Education. REL Appalachia researchers in rural education, early warning systems (EWSs), data use. What Is the VMSRA? 9 – Louisa County – Norton City – Page County – Pulaski County – Russell County – Rockingham County – Smyth County – Staunton City – Tazewell County – Wythe County

10. VMSRA Goals Assist middle school practitioners in using data to inform instructional decisionmaking and improve student outcomes by: – Identifying struggling students who need additional support. – Selecting, implementing, and monitoring interventions to support students. REL Appalachia’s work with the alliance will focus on expanding the state’s EWS to middle schools. EWS implementation is taking place beginning with four interrelated projects in 2014/15. 10

11. What Is an EWS? Provides a systematic way to identify (“flag”) students early who are at risk of failure. Grounded in research. Relies on readily available (and familiar) data: ABCs – Attendance, Behavior, and Course grades/assessment results. Provides information that is actionable by educators and other adults in schools and divisions. Requires educators to diagnose further student needs, and to use professional judgment to support at-risk students. Targets resources to support at-risk students while they are still in school, before they go too far down the road of academic failure and drop out. Examines patterns and identifies school climate issues. 11

12. Why Middle School Matters A student’s decision to drop out of school is a gradual process that starts well before high school.* – The habits and patterns set in middle school follow students to high school. – Only 50% of students entering high school two or more years behind grade level in math/literacy are promoted to grade 10. – Students repeating grade 9 have a 20% likelihood of graduating from high school. Students at risk of dropping out can be identified as early as grade 6 and share characteristics of low attendance rates, course failure, and low assessment scores.** An EWS helps catch these students before it’s too late. * Allensworth & Easton (2005, 2007) ** Balfanz (2009); Balfanz, R., & Herzog, L. (2005) 12

13. How Does an EWS Work?

14. The EWS Continuous Improvement Cycle Step 1: Establish roles and responsibilities Step 2: Use the EWS Middle Grades Tool Step 3: Review the EWS data Step 4: Interpret the EWS data Step 5: Assign and provide interventions Step 6: Monitor students and interventions Step 7: Evaluate and refine the EWIMS process 14 * Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013)

15. Infrastructure and Support is Just as Important as the Process

16. EWS – A Systems Perspective Step 1: Establish roles and responsibilities Step 2: Use the EWS Middle Grades Tool Step 3: Review the EWS data Step 4: Interpret the EWS data Step 5: Assign and provide interventions Step 6: Monitor students and interventions Step 7: Evaluate and refine the EWIMS process 16 * Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M., (2013) A.Data and Computer Data Systems B.Educator Knowledge & Skills for Data Use C.School Organization for Data Use

17. Projects Toward EWS Implementation

18. VMSRA Projects Data catalog of assessments and interventions currently being collected in schools. Document how data are currently being used in schools, and what supports and barriers are in place to help or that hinder use. Identify most powerful data to track.* Provide workshops on using data efficiently and effectively to:* – Identify and target resources to struggling students. – Monitor each student’s progress toward mastery. * These are proposed activities and not yet approved by IES. 18

19. What Have We Learned So Far? Most data are available for an EWS; using data is difficult. – Extracting or “triangulating” multiple data points is complex. – Data are not always centrally located in one program. – Not all pertinent data are electronic; some are kept in hard copy only. Analyzing historical patterns for students who are struggling may not be possible because some data do not “save” from year to year. Sharing information about students across school levels—from elementary to middle to high school—is not common. Assessment data is not always teacher-friendly: working with Alliance to balance assessments FOR learning with assessments OF learning. 19

20. Difference Between Assessments OF and FOR Learning CriteriaAssessments OF LearningAssessments FOR Learning PurposeCheck achievement status for public reporting Help students learn more InformsOthers about students: teachers, school leaders, policy makers, etc. Students about themselves Focus onState or local standardsAchievement targets that underpin standards Answers questions Have students MET standards?Have students mastered the learning blocks/foundations of the learning standards? TimingAfter learning is supposed to have occurred During learning, while there is still time to help UsesCertification; public reporting; grading; gatekeeper decisions Understand success; diagnose needs; see progress 20 Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg

21. Difference Between Assessments OF and FOR Learning CriteriaAssessments OF LearningAssessments FOR Learning Teachers role TimekeeperPartner; coach Students role Get the highest score; avoid failureUnderstand success; do better next time MotivationReward and punishmentBelief that success is within reach if they keep trying 21 Adapted from Stiggins, R. http://www.youtube.com/watch?v=EDVHuHyCGmg

22. Importance of a Balanced Assessment Portfolio to an EWS An EWS uses historical patterns to identify students in need of further assistance. It’s up to educators to get students in need of assistance back on track. Educators need the right data. – To meet student’s needs, educators have to know what those needs are. – To determine whether interventions are working, educators need to know what to monitor the progress of, and they need data to be able to do the monitoring. Assessments OF learning help determine whether a student is struggling. Assessments FOR learning help determine how to help the student. 22

23. Discussion What is the one thing we must get right to make an EWS worth undertaking? What are current hurdles or barriers that would make implementing an EWS difficult? Acting on data difficult? Where does an EWS help meet your goals in your division? Where does it fall short? 23

24. Future Work with the VMSRA Data-use interviews: How are you using data? What supports exist? What barriers hinder? Validation study*: What are the most powerful data to identify struggling students? Workshop for data leaders* (July 23): superintendents, assistant superintendents, curriculum specialists, directors of assessment/accountability, building principals. – What does effective data practice look like? – How do you create an entrenched strategy around using data, from the division to the classroom? What foundational elements have been found to support effective data use? What additional/different elements are needed to use data effectively? Workshops* to assist in creating or building on these foundational elements. 24 * These are proposed activities and not yet approved by IES.

25. Alliance Membership Review Open to any Virginia middle school, district, or state officer: – Principals and central office staff are greatly encouraged to join. Choose a membership level (core or general) at your discretion; time commitment varies: – Core group will participate in monthly phone conferences with REL Appalachia to develop the research agenda and project implementation plans for 2013/14. Thereafter, calls will be bi-monthly to discuss project progress, receive information, or offer feedback. – General members will participate in quarterly meetings to receive information and offer feedback on alliance projects. All members may choose to participate on alliance project teams or advisory committees. REL Appalachia staff will coordinate and manage all REL work.

26. References and Materials Allensworth, E. M., & Easton, J. Q. (2005, June). The on-track indicator as a predictor of high school graduation. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf Allensworth, E. M., & Easton, J. Q. (2007, July). What matters for staying on-track and graduating in Chicago public high schools: A close look at course grades, failures, and attendance in the freshman year. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf Balfanz, R. (2009). Putting middle grades students on the graduation path: A policy and practice brief. Baltimore: Johns Hopkins University, Everyone Graduates Center. Retrieved from http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policy- and-practice-brief/ ‎ http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policy- and-practice-brief/ Balfanz, R., & Herzog, L. (2005). Keeping middle grades students on-track to graduation: Initial analysis and implications. Presentation at the second Regional Middle Grades Symposium, Philadelphia, PA. Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M. (2013). Middle grades early warning intervention monitoring system implementation guide. Washington, DC: National High School Center at American Institutes for Research. Retrieved from http://betterhighschools.org/EWS_middle.asp http://betterhighschools.org/EWS_middle.asp 26

27. Connect with Us! www.relappalachia.org @REL_Appalachia Aimee Evan aevan@QRA-LLC.net 703-655-3695 27