1. Module 1
EWS Data Literacy
Module 1
EWS Data Literacy

2. Core Ideas of Early Warning Systems
To graduate, college- and career-ready, students need to successfully navigate several key transitions and acquire a set of academic behaviors - they need to learn how to succeed at school.
Students signal that they are on- or
off-track toward these outcomes
through their behaviors
There are four core ideas to Early Warning Systems that will be important as you examine implementation at your school. The first two are posted here.

3. Core Ideas of Early Warning Systems
By tracking Early Warning Indicators, it is possible to identify when students are beginning to fall off-track, providing time to intervene and alter their trajectory through school and beyond.
Using EWS, schools can be organized to apply
school-wide preventative, targeted
and intensive interventions until
students are on-track.
There are four core ideas to Early Warning Systems that will be important as you examine implementation at your school. The last two are posted here.

4. Facilitation and Teams
Early Warning Systems
Indicators
Support and response system
Reliable, valid, and predictive
Each indicator adds information for action
Easily accessed and organized
Whole school/classroom, small group, and individual initiatives and interventions
System for monitoring/follow-up
Reflective processes
The Early Warning System combines the data system that accurately identifies students, has teacher access to student information, and the intervention system that identifies the appropriate resource and is able to act in a reasonable amount of time. Some of these are human elements and school structures, while others may be technology or data systems. Both are essential to make this system work.
This part of the presentation will focus on the early warning data systems and how to use the information with school staff to monitor student progress, identify patterns/trends, and improve practices.
Facilitation and Teams

5. Module 1 – EWS Data Literacy
Part 1: What are early warning indicators of student success and why are they useful? (≈ 20 minutes)
Part 2: The ABCs (≈ 20 minutes)
Part 3: Looking for patterns in data (≈ 20 minutes)
This module provides fundamental information about early warning indicators. It includes information from research findings about these indicators of future student success. It presents how the data can be organized to be most useful and particular information that will be helpful for your team to consider as you examine student data. This module has three sections. The first focuses on what are early warning indicators, the second section goes into more depth on each of the indicators and their implications from research, and the last section demonstrates how to analyze patterns in the early warning indicators’ data.

6. Module 1 – Part 1
What are early warning indicators of student success and why are they useful?

7. Characteristics of Early Warning Indicators
Are reliable and valid-They flag students who without effective intervention would not have a good result.
Are practical and useful for school personnel- For example, they identify a significant number of students who without effective intervention have high odds of dropping-out.
Are selective- Each indicator adds additional and actionable information that is not provided by another indicator.
Research about early warning indicators has focused on finding student behaviors or characteristics that provide reliable and valid indicators of future student success that are practical and useful for staff members, and provide actionable information.

8. Determining On- and Off-Track Indicators
9th Grade
10th Grade
11th Grade
12th Grade
Expected Graduation
Research Based
Need Intervention
High Probability
Majority of Students
Off-track
On-Track
Off-track indicators are found through researching student records and are identified by following cohorts of students over time. For example, studies have followed students from when they entered the 6th or 9th grade until the year they should have graduated from high school and looked for commonalities among students who did not graduate at the expected time. Students with off-track indicators have low odds of achieving a desired outcome (e.g. high school graduation or postsecondary enrollment) unless they receive effective interventions. At a minimum, off-track indicators should indicate that the probability of students achieving a desired outcome is less than 50%. The higher the odds, the more efficient the indicator. Strong indicators also identify a substantial portion of students who do not achieve a desired outcome.
On-track indicators have similar properties to off-track indicators but they indicate that students who possess them have high odds of achieving a desired outcome (e.g. post-secondary success).

9. On and Off-Track Indicators
On-Track
High School Graduation
On-Track
Postsecondary Success
Sliding
Based on what has been learned in research, the terms on- and off-track are used to help identify students earlier who may need additional support. On-track indicators are the behaviors of a student who meets the predictive criteria to graduate from high school and/or succeed in college or future careers. Research has found that students who achieve at high levels in these indicators have a better chance for high school and postsecondary success. This has really shown that high achievement matters in each of these indicators. Thus, the focus should be on both ensuring that students do not fall off-track and pushing them towards higher levels.
Many schools have added a sliding category to help them catch a problem before it gets too large. For instance, many schools where failing a course is an off-track indicator, have now made getting a D in a course a sliding indicator so that they are able to intervene earlier.

10. EWS Lessons from the Field: Early Warning Flags
Indicate that a student is moving towards off- track status or away from on-track status
Are research based, but also need to be grounded in common sense
For example, if attendance, behavior, or course performance dips it is important to know why and to address it quickly
By adopting these early warning flags, which actually predict high and low achievement, we are able to determine where a student may be headed unless they receive intervention. These early warning flags also need to be practical and useful in order to provide value for school staff members or partners who are responding to student needs.

11. EWS Lessons from the Field: Organizing
Be Selective: Center efforts around a few high yield indicators.
Provide Early Response: Design systems that respond to student behaviors before triggers for the more intensive interventions are reached.
Put Students First: Rapid identification is important but so is designing interventions that are built on student strengths.
Lessons from early implementers of EWS have helped to see key particulars. Each of these three specific lessons have helped make the research about early indicators more practical for those implementing and supporting students.

12. Why EWS Matters: What has been Learned from Research
Students in high-poverty schools who successfully navigate grades 6 to 10, on-time and on-track, by and large, graduate from high school.
Students in high-poverty schools who struggle and become disengaged in the early secondary grades and in particular have an unsuccessful 6th- and/or 9th- grade transition do not graduate (often 25% or less graduation rates).
The 6th – 9th grade time period is important. Research has shown that in high poverty environments, we can identify up to 50% of eventual dropouts in the 6th grade and up to 75% of dropouts by the end of the 9th grade, giving a very accurate picture of students who need additional support.
When students are able to manage this transition and stay on-time and on-track, their chances for secondary and postsecondary success increase greatly.
See research studies under Module 1 resources

13. Applying an EWS: Students Change at Different Grade Levels and Require Different Supports
Pre-K and Elementary Grades- Core academic competencies and socialized into the norms of schooling in a joyful manner
Middle Grades- Intermediate academic skills (reading comprehension and fluency, transition from arithmetic to mathematics) and a need for adventure and camaraderie
High School- Transition to adult behaviors and mind set with a path to college and career readiness, as well as, the appropriate extra support for students with below grade level skills
There are significant transitions where early warning indicators become even more important. They follow key developmental periods in a student’s life as well as school structures where additional challenges may arise.

14. Activity – Mini-quiz
What are the four important characteristics that define an off-track indicator?
What are the key transitions (grade levels) for students?
What are the differences between on- and off-track indicators?
Why is the above information (answers to 1, 2, 3) important for your school?
Please take a few minutes individually or with your group to complete the mini-quiz. These questions will help you reflect on the information from the previous slides and refresh your memory on the key points of early warning indicators of student success and why they are useful.

15. Attendance, Behavior, Course Performance- The ABC’s
Module 1 – Part 2
Attendance, Behavior, Course Performance- The ABC’s

16. Indicators and Influencers?
Behavior marks
Suspensions
Poor attendance
Overage: 1-2 years, 2 years+
Special education English Language Learners Standardized test scores
Course failure
Core courses
Elective courses
As discussed in the previous section, researchers followed a cohort of students to try to determine which behaviors or characteristics were the most predictive of students likely to fall off-track and leave school. It was particularly important to look for things that were not only influencers, as many of these items are, but were also able to make predictions about a majority of students. Of these items, there are three that have continued to rise to the top as key indicators.
Pause and Discuss: What do you think are the 3 key predictors of student success?
Gender
Socio-economic status
Parental education
Poor behavior
Repeaters
9th graders

17. Most Consistent Predictors of Student Success are the ABC’s
Student Engagement B A C
Attendance
Behavior
Course
Performance
Attendance, behavior, and course performance/credit accrual are the three best and most consistent predictors of future student success. Later slides will examine each of these predictors with key findings that reflect the relationships between the ABC’s and student outcomes.
Pause and discuss, “Why do you think these are the most important predictors of student success?”

18. The Primary Off-Track Indicators for Potential Dropouts:
For Half or More of Eventual Dropouts the Path to Exiting School Without a Diploma Begins as Early as the 6th Grade
The Primary Off-Track Indicators for Potential Dropouts:
Attendance: < 80% school attendance
Behavior: “unsatisfactory” final behavior mark in at least one class
Course Performance: A final grade of “F” in Math or English
This study, conducted in Philadelphia examined all 6th grade students in and followed those students through to , which was one year past their normal expected high school graduation date. What’s evident is that only 10-20% of the students who were identified with any one of these indicators at the end of 6th grade were able to graduate in this time period. As was previously mentioned, the 6th – 9th grade transition is important because only about 35-40% of these students make it to 10th grade on-time. In the Philadelphia study, the attendance rate was less than 80% and that was an indicator of students who rarely graduated. However, other cities’ and states’ studies have found that the key point at which students fell off-track for attendance ranged from 85% to 90%.
It is important that the exact level of indicators be determined based on your own school data.
Sixth-grade students in high poverty environments with one or more of the indicators may have only a 10% to 20% chance of graduating from high school on time or within one year of expected graduation.
Source: Balfanz and Herzog, Johns Hopkins University and Philadelphia Education Fund, 2007
See research studies under Module 1 resources .

19. What Do we Know about Attendance Indicators?
Students who miss 10% or more days in a school year need intervention.
Students who miss 5 or fewer days in the school year thrive.
Some attention should also be paid to students who miss more than 5 days but less than 10% (i.e. check-in and monitor) because they have the potential to slide off-track.
Compiled research has shown that students who miss 10% or more of school (which is about 180 days) in a full school year need intervention. However, high levels of attendance are essential to student success, so it is important to look at students missing 10% or more, while pushing for more and more students to miss 5 days or less.

20. Impact of Attendance on High School Graduation and Post-Secondary Enrollment
Attendance is the first indicator and research studies have continued to find that attendance is predictive of a student’s future outcomes. The chart above shows the progress of 10th graders through high school graduation and enrollment in post secondary schools. Students with less then 9 absences were found to graduate from high school and enroll in postsecondary at much higher levels than those with 10 or more absences.
Source: Education Longitudinal Study of 2002 (ELS:2002)
See research studies under Module 1 resources

21. Impact of Attendance on Standardized Test Scores
Lack of attendance has a negative impact on other school outcomes as well, including achievement. The chart above shows the results from a group of Florida students. As you read from left to right in the chart, you can see that as the student number of absences increases, their FCAT score (state assessment) continues to drop.
Source: Sent Home and Put Off-Track, Balfanz, Fox, and Byrnes, 2012
See research studies under Module 1 resources

22. Focus on the ABC’s-Attendance
Schools and communities need to measure and act on chronic absenteeism
# of students who miss 10% or more of school
# of students who miss a week or less
Organize efforts built around knowledge that student absenteeism is driven by a combination of:
Student choice, school factors driving students away, and out-of-school factors pulling them away
Attendance is often measured by an average daily attendance rate (ADA), which gives a good general measure of attendance at a school but hides some key information and potentially large numbers of students. A school can have 92% ADA and yet 20% of its students are missing a month or more of school. Chronic absenteeism (students missing more than 10% of school) should be measured alongside average daily attendance to get an accurate measure.
Student and staff interviews have shown that students are not attending for a variety of reasons and support structures must be adjusted and designed to meet these needs.

23. Focus on ABC’s-Attendance Let’s Make Students Excited about Coming to School
Most engaged secondary grade students are involved in cognitively rich activities which combine teamwork with performance (robotics, debate, drama, etc.)
Build an attendance problem solving capacity which may involve external partnerships who can provide supports for the out-of-school challenges that students face.
Whether students are choosing to step away, are being pulled away, or are being pushed away, schools can play a role in making school an engaging place to be and providing supports that help students address some of the outside challenges students are facing.

24. What Do We Know About Behavior Indicators
First, it is important to track minor as well as major incidents.
Students with sustained mild misbehavior fall off-track in large numbers.
Behavior is tracked differently across schools, districts, and states giving some variation in findings. However, there have been consistent results that major issues (often measured by suspension) are of concern but that sustained mild misbehavior is aa equally strong predictor of students who will fall off-track in the future.

25. We Need Alternatives to Suspensions
Students with one or more suspensions need intervention to make sure they are not suspended again and/or fall off-track.
In Florida, being suspended in the 9th grade once, increased the odds of dropping out; being suspended two or more times greatly increased the odds.
Source: Sent Home and Put off-Track, Balfanz, Fox, and Byrnes, 2012
There has been evidence that more serious incidents of suspension have great impacts on a student’s chances of graduation.

26. Focus on ABC’s- Student Effort Also Matters
In Chicago, student effort was found to have greater impact on course passing then demographic variables.
Source: What Matters for Staying on-Track and Graduating in Chicago Public Schools, Allensworth and Easton, 2007
ACT tested over 14,000 students at 48 postsecondary institutions. Results show that motivation, social engagement and self-regulation are key factors in students’ success in high school and college.
Source: Enhancing College and Career Readiness and Success: The Role of Academic Behaviors, Robbins, Allen, Casillas, Peterson, & Le, 2006; Allen, Robbins, Casillas, & Oh, 2008
In the Chicago work, they found that a student‘s effort and what they termed academic behaviors had more impact on course passing than any demographic variables.
ACT Engage College and Career Readiness System provides additional information that more accurately identifies students who are at risk of poor grades and drop out, beyond measures of academic achievement alone.

27. Focus on ABC’s-Behavior and Effort
Model and teach resiliency, self-management and organization skills.
Implement school-wide positive behavior support programs and alternatives to suspensions.
May need to re-examine disciplinary policies
Work to insure that students’ experience consistent academic and behavioral norms as they travel from class to class.
Need adults to look at what they can do to make students successful.
From early implementation of EWS, responses to behavior need to examine school-wide practices (e.g., discipline policies, practices for make-up work) as well as support systems (e.g., whole school campaigns, initiatives, and modeling).
Many students who have struggled in school or seen struggles at home disengage from their work instead of trying in their assignments. This disengagement doesn’t give them an opportunity to learn.
Pause and discuss: What support systems and programs does you school have? Are there others that you would find helpful to support positive school behavior? Are there others that you would find helpful to model and teach good behavior?

28. What Do We Know About Course Performance Indicators?
Course performance is as important for students long term success as test scores.
6th and 9th graders with multiple D’s and F’s seldom graduate unless interventions are applied and take place consistently.
6th and 9th graders with B or better averages typically succeed in high school and are on-track for future college success.
The third indicator, course performance, is especially important. Course performance has been shown to be a better predictor of long-term success (both in secondary and post-secondary work) than test scores, GPA, and credit accrual. And importantly, similar to attendance, high achievement matters. It is essential for students to not receive F’s but it is as important that we push and support more students to receive A’s and B’s.

29. Focus on ABC’s- Course Performance
Course performance is comprised of a wide variety of things ranging from assignment completion, preparation for tests and quizzes, prior preparation, and classwork.
Students need support and sometimes even advocacy to succeed in their learning, as well as, tutoring, mentoring and other support programs.
Schools should offer effective second chance and credit recovery programs which hold students accountable but provide a reason for them to keep trying.
There are a variety of reasons why students do not achieve at high levels in courses. There needs to be supports to help them succeed and if necessary, recover after they have failed courses.
Pause and discuss: What are the reasons that students fail courses? What supports do you have at your school to help students be successful in their courses?

30. Focus on ABC’s – Course Performance Course Credit
The courses that a student passes (i.e., what subjects) becomes more important in high school.
Credit accumulation should be examined alongside course passage.
As students enter high school, course performance increasingly needs to cover examination of credit accumulation. This is important for a number of reasons.
First, specific core courses, especially in the ninth grade form the basis of the credits needed for graduation. Students need to be passing math and English but as importantly accumulating the necessary credits to move from 9th to 10th to 11th to 12th grade.
Second, many ninth grade core courses are required in order to take the second level of the same subject area, thus delaying a student’s opportunity to earn other credits if they have not passed core courses.

31. 6th – 9th Grade Window 6th Grade On-track and On-Time 9th Grade
The 6th – 9th grade window is the focal point for ensuring students remain on track. In each of these indicators, schools and communities that help their students make it through this time period on-time and on-track, greatly increase the student’s chances of secondary and postsecondary success. In some cases, this has been estimated to be a three fold increase, from 25% chance of secondary success if off-track by the end of 9th grade increasing to a 75% chance of secondary success if on-track and on-time by the end of the 9th grade.

32. Activity: Examining the Data
There are four slides that contain ABC data to examine.
(If you have a group of four, have each person
examine one and then share with the group.)
What is each data slide showing?
What are the implications for your school?
What does it say?
As an activity, you will be a looking at a few research findings to determine what takeaways you see for implementing EWS at your school. There are four slides of data for you to examine (discuss with your group) the implications for your school. We recommend that you divide into four groups, each will examine one piece of data, and then have each group share their findings.

33. HS Freshman Grades Matter
Virtually all students with a “B” avg. or higher graduate in 4 years
Prediction is less certain among students with D+, C- , C
Virtually all students with less than a “D” avg. fail to graduate
This chart is for the Activity: Examining the Data.
Overview:
The chart shows the GPA of Freshman students and the percentage of these students who graduated within four years. Students with a Freshman GPA of 0 or .5 (D- or F) graduated at rates under 10% while students with (B+ to A) graduated at rates over 90%. Students in the middle ( ) were much more mixed.
What Matters for Staying On-Track and Graduating in Chicago Public High Schools, Allensworth and Easton, Consortium on Chicago School Research, 2007

34. What Matters for Staying On-Track and Graduating in Chicago Public High Schools, Allensworth and Easton, Consortium on Chicago School Research, 2007
This chart is for the Activity: Examining the Data.
Overview:
This chart shows four year graduation rates for students based on the number of days that a student missed in a semester. Students missing more than 10 days graduated at less than 50% rates. The students who missed four or fewer days graduated at high levels (87%).

35. Los Angeles Course Failure
What Factors Predict High School Graduation in the Los Angeles Unified School District? Silver, D., Saunders, M. (University of California, Los Angeles), Zarate, E. (University of California, Irvine)
This chart is for the Activity: Examining the Data.
Overview:
The chart shows the timing of a failed class in middle school affects the chances of graduating.
Students who fail classes in 6th grade or 7th grade only, but recover (that is, fail no classes
in 8th grade), graduate from high school at much higher rates than those students who
continue their pattern of failing courses. Additionally, students who don’t fail classes
until high school further improve their chances of graduation.

36. Only 17% graduate on time or within one extra year.
Sixth graders with poor behavior (earning an unsatisfactory final behavior mark) have a 1 in 4 chance of making it to the 12th grade on time.
Only 17% graduate on time or within one extra year.
This chart is for the Activity: Examining the Data.
Overview:
The chart follows 1660 students who had an “unsatisfactory mark for behavior” on their 6th grade report card from 6th grade until when we would expect them (based on their age) to graduate from high school. 73% of these students made it to 9th grade with their peers but only 34% made it on to 10th grade. By the time of graduation, only 7% of these students had graduated at the expected time.
Source: Balfanz and Herzog, Johns Hopkins University and Philadelphia Education Fund, 2007

37. Looking for patterns in Data
Module #1 – Part 3
Looking for patterns in Data

38. Understanding Patterns
To understand patterns in attendance, behavior, and course performance, the EWS indicators need to be studied and monitored.
Ways to look at EWS data:
- Individual student patterns
- Class, grade, school patterns
These variant patterns can lead to different solutions and interventions.
Pause and discuss: Why is it important to look at the different patterns in the EWS data?

39. Examining Patterns
At the student level (over time and across indicators)
Across the ABC’s
Across students
Attendance
Behavior
Course Performance
Off-Track
Less than 90%
1+ suspension and/or mild sustained misbehavior
Failing ELA and/or Math
On-Track
Greater than 90%
No suspensions or mild misbehavior
Passing ELA and/or Math
College Ready
Greater than 95%
B or Better
In order to begin examination of attendance, behavior, and course performance data, a full student profile is necessary. In this section of the module, a few important ways to examine ABC data will be discussed, helping to deepen the examination of trends.
For these patterns, the above table is used to categorize the actions as on- or off-track.

40. 4 Diagnostic Questions About Students
Are they regularly attending school? If not, why not?
Are they able to focus on schoolwork in school?
Are they productively persistent, i.e., trying in an effective manner?
Do they connect school effort to life success?
As you look at student level data, these four questions are guidance to determine the best areas for response.

41. Examine Individual Student Data
Attendance
Behavior (referrals)
Course Performance
13-14 year
Sept
Math Q4
Math Sept.
ELA Q4
ELA Sept
#John
96%
5 days 7 B
68% A
77%
#Adrian
93% 2
4, 1 Susp.
88% C
53%
#Samuel
99%
0, 10 Tardy 14
82% D
55%
#Erica
81% 5 1
56%
87%
For each student, ask some basic questions to better understand each student and be better prepared to respond to their needs.
Are there trends up or down in any one indicator over time (last quarter, previous school year, etc.)?
Is the student off-track in one or more than one indicator? Which ones?
Are there major differences between the student’s performance in indicators that give you an idea of a student’s strengths (e.g., strong in academic performance but low in attendance)?
What additional questions do you have to learn about this student? How could you find out?

42. Looking for Patterns at Student Level
Are there trends up or down in any one indicator over time?
Is the student off-track in more than one indicator? Which ones?
Is there any relationship between indicators?
Are there major differences in indicators that give you an idea of a student’s strengths (for example, strong in academic performance but low attendance)?
What additional questions do you have to learn about this student? How could you find out?
Pause and discuss these questions about the previous slide, titled ”Examine Individual Student Data.”

43. Sample Classroom Level Data Display- Off-Track Indicators
For an individual student, the first level is looking across time periods to see if there is progress for a student. In this example from Pennsylvania’s statewide system, you can see code of conduct incidents, Math and English Language Arts Grade, and 3 columns looking at attendance trends.
This look across time helps to gauge which direction a student may be heading and how you can intervene or accelerate their progress. The red, green, and gray triangles in this example help show whether a student is improving, declining or not having any change. 43

44. Off-Track Indicators Plus Student Survey Resiliency Data
When looking across a number of indicators or factors, an additional set of questions arise. The example here shows student ABC data with the addition of results from a resiliency survey where a student responded to questions about a series of important factors that often show up prior to being off-track in the ABC’s.
Are there areas of mismatch in the student data that give an idea of student strength (doing well in one area but not in others)?
Are there items showing up in resiliency factors that are not showing up in the ABCs?
Does it look like attendance is impacting course performance?
Does it look like behavior is impacting attendance (suspensions or other) or course performance?
This additional information can help identify patterns in order to help the team make better decisions about the types of responses or interventions that have the best impact for a student.

45. Now let’s look across multiple students
Now let’s look across multiple students. This becomes especially important as you examine proactive and small group strategies to prevent more students from falling off-track and to make whole classroom or whole school adjustments to support more students.
Pause and discuss:
Is there one indicator that is particularly strong or weak?
Are there any correlations between different indicators?
Are there groups of students who may be facing the same challenge?

46. Activity
For this activity, examine the class of students that has been provided and, as an individual, or group answer the following questions:
What trends, patterns, or ideas do you notice?
What might be occurring with these students?
If these were a class in your school, what would be your response to this data?

47. Follow-up to Activity
You may have come up with a number of items from the data that were shown and were able to identify responses that could occur in your school. Given the data that was shown, it’s anticipated that most schools will identify a mixture of individual responses for particular students (e.g., the third student from the bottom seems to have unique flags.), for small group responses (e.g., two students who are possibly having attendance problems), and for whole school or classroom initiatives (e.g., is something occurring in science class). Seeing these trends as both individual items to address as well as items better addressed across groups of students will help your school be proactive about challenges and more efficient in what you are able to address.

48. You can now do the same activity with a sample class from the PDE EWS tracker. You will notice here that you have access to the ABC’s and the red, green, and gray triangles indicate if there is improvement, decline, or no change.
What patterns or data do you notice in this group of students?

49. In the previous slides, we have used sample data sets to help you see all of the data components that go into an Early Warning System. In the Pennsylvania Educator Dashboard and Early Warning System, you will be able to access similar reports. This screen shot is a sample of a class of students and their performance in the ABC’s. As you notice, the red and green show on- or off-track as well as arrows to show change over time.