Reflective Assessments Participants should go to the EVAAS wiki. Under the agenda section have users click on the reflective assessments link. Files to support this portion of the presentation can be found there.

Value-Added Reports Use to evaluate the overall effectiveness of a school on student progress. Compares each school to the average school in the state. Comparisons are made for each subject tested in the given year and indicate how a school influences student progress in those subjects. Has to be more than -1.8 to be below; more than 2 standard errors to be above. (Provide definition of value-added.) Some things to note are that “like” students are in a subgroup. Show how to read the report. Explain that 0 is the equivalent of one year of growth. Looking at the bottom of the page at the green, yellow, and red descriptors. Explain that there is a blue descriptor when there is not enough data to make a distinction. If your LEA uses DIBELS, the reports have similarities in design with the colors. Have the participants look at the data and talk about how you can see here. Look at trends in the same grade. Look at the students that moved from 6th grade in 2011 to 7th grade in 2012 and to 8th grade in 2012. Talk about what you can take a way from this report.

Diagnostic Reports – the whiskers In this diagram, the two bars have the same height, but the whiskers extend to different lengths. On the left, the whiskers lie completely below the green line, so the group represented by the bar made less than average progress (↓). On the right, the whiskers contain the green line, so the group represented by the bar made average progress (-). The red whisker represents the confidence interval due to the standard error for the mean.  There is a high probability that the actual mean falls somewhere within the Whiskers.  The size of the confidence interval is determined by the sample size.  The larger number of data points the smaller the standard error, smaller the whisker, and therefore the confidence interval.   If the whisker passes over the green line (reference line),the data shows expected growth since there is a chance the mean is actually on the other side of the green line.  It is not certain that the teacher is exclusively above or below the reference line, if the whisker crosses the green reference line. 

Diagnostic Reports Looking for Patterns In this school, some subgroups of students are not making sufficient gain. Students in the lowest subgroups have not made sufficient progress while high achieving students are making excellent gain. The lack of appropriate progress among low achieving students is a pattern that has been repeated from previous years, indicating a persistent lack of effectiveness with lower achieving students. This is one of the most intriguing components of EVAAS. Common Diagnostic Patterns activity *Pattern Slides are on wiki

School Diagnostic Shed Pattern In this example, the lowest achieving students are making sufficient progress. Students at an average achievement level are making expected progress. However, the highest achieving students appear to be losing ground. Teachers and administrators will want to find ways to create more progress opportunities for high achieving students.

School Diagnostic Reverse Shed Pattern In this example, high achieving students are making excellent progress. Students who are average in achievement also are making sufficient progress. In contrast, the lowest achieving students are not making as much progress as they should. A pattern like this one will widen the achievement gap. Teachers and administrators should consider how to help lower achieving students gain more ground.

School Diagnostic Tent Pattern In this example, the students in the middle of the achievement distribution are making sufficient progress, but both lower achieving and higher achieving students are falling behind their peers. In this case, teachers and administrators will want to consider both how to support low-achieving students and how to challenge high-achieving students.

School Diagnostic V Pattern In this example, the opposite of the Tent Pattern, only the lowest and the highest achieving students are making good progress. Students in between have not had enough opportunities for academic growth.

School Diagnostic Opportunity Gap Pattern In this example, the students in every achievement group are making sufficient progress in the most recent year, except for the second group. Teachers and administrators will want to consider how to adjust the classroom instruction to meet these students’ needs. In addition, what approaches that are successful with the lowest achieving students could be expanded to include students in the second achievement group?

What would an ideal pattern on a Diagnostic Report look like for closing the achievement gap? Have participants draw an ideal pattern On index card, make a box with 1-5 at bottom. Draw the ideal diagnostic report; discuss Ideal to narrow the achievement gap: 1 is highest, descending to 5 Common Diagnostic Patterns activity – look at common patterns; taking turns, explain the pattern to your partner

Diagnostic Reports – Desirable Pattern Print and handout this slide for the next activity on drawing a desirable pattern or have participants use plain paper.

Diagnostic Report Desirable Pattern In this example, all bars above the green line indicating the district was highly effective with students in all achievement groups. Additionally, students in the lowest quintile made more progress than students in the other quintiles. Effectively, these students are starting to catch up with their peers; the gap is closing because they are increasing their performance more than a year’s worth of growth.

Diagnostic & Performance Diagnostic Reports (Part 2)

Overview of School Effects (sample data) Place activity instructions on the wiki. Use the Value-Added and Diagnostic reports to complete the table. Navigate to a Value-Added Report and enter the Tested Subject/Grade name in the Overview of School Effectiveness table below. Locate the color for the most recent year. If the color is RED, place an “X” in the Overall Results column. Use a separate row for each grade for EOG reporting. If your school tests in both EOG and EOC subjects, record the EOG subjects and grades and then choose EOC from the Tests tab. For each test, subject, and/or grade, note the color for the most recent year. If the color is RED, place an “X” in the Overall Results column.

Overview of School Effects (sample data) Drill down to the Diagnostic Report for each Tested Subject/Grade. Locate the blue bars on the graph for each of the 5 Achievement Groups. Also note the red whiskers. For any blue bars above the green line (where the whiskers are also completely above), place an up arrow () in the appropriate cell of the Overview of School Effectiveness table. For any blue bars below the green line (where the whiskers are also completely below), place a down arrow () in the table. For any blue bars at or near the green line (the whiskers cross the green line), place a horizontal dash (–) in the table.

Overview of School Effects (sample data) These documents will be uploaded to the wiki An extra set of sample data has been loaded on the wiki if participants can’t log into EVAAS

Overview of School Effects (sample data) These documents will be uploaded to the wiki

Overview of School Effects (sample data) These documents will be uploaded to the wiki

Overview of School Effects (sample data) Double check for correct answers

1. Go to the website www.ncdpi.sas.com Copyright © 2010, SAS Institute Inc. All rights reserved.

1. Go to the website ncdpi.sas.com Copyright © 2010, SAS Institute Inc. All rights reserved.

1. Go to ncdpi.sas.com 2. BOOKMARK IT! 3. Secure & Convenient Online Login Copyright © 2010, SAS Institute Inc. All rights reserved.

Do you see this? Then Sit Tight! Copyright © 2010, SAS Institute Inc. All rights reserved.

Overview of School Effects It’s Your Turn! Find the blank table. Do this by yourself. Using sample data Fill in your table. Participants will examine sample data located on wiki site to complete this activity. Locate the blue bars on the graph for each of the 5 Achievement Groups. Also note the red whiskers. For any blue bars above the green line (where the whiskers are also completely above), place an up arrow () in the appropriate cell of the Overview of School Effectiveness table. For any blue bars below the green line (where the whiskers are also completely below), place a down arrow () in the table. For any blue bars at or near the green line (the whiskers cross the green line), place a horizontal dash (–) in the table.

Overview of School Effects What did you find? Interesting Patterns Insights Areas of Concern Areas of Celebration This is a sharing activity – Think-Pair-Share, TTP, Tea Party, etc.

1. Go to the website ncdpi.sas.com Log back in. Copyright © 2010, SAS Institute Inc. All rights reserved.

Finding Your Patterns Have participants find their school or a school in their district. Identify the patterns for your subjects and grade levels Patterns and explanations can be located on the wiki in the Reflective Assessments portion of the agenda.

Interpreting Your Results Complete the Interpreting Your Results downloadable take home form on the Wiki. Have participants go to the Reflective Assessments section of the wiki and download the Interpreting Your Results documents.

Student Pattern Report This report is a customized Diagnostic report where you can examine progress for groups of students of your choice. It is only available from the school level and only to users w/access to student reports.

Student Patterns Report Key points to remember: The report shows growth for the lowest, middle, and highest achieving students within the chosen group. The report can be used to explore the progress of students with similar educational opportunities. Like all diagnostic reports, this report is for diagnostic purposes only. A minimum of 15 students is needed to create a Student Pattern Report. Enables you to see how effective the school has been with lowest, middle and highest achieveing students at least 15 w/predicted and observed scores.

Student Pattern Report Take a look at this data, what do you notice? What are your thoughts? Higher students did better than expected. Our ML did not do as well as predicted The groups l, m, h placed in 3rds based on their predicted scores and where they fall in the distribution Teacher self reflection how effective were you in teaching based on their predicted score.

Student Pattern Report This teacher may have had 100% proficiency but we are looking at growth. This teacher did not contribute to the students learning. Students listed by name by subgroups, so that you can identify which group each child falls in. Look at individual students to see if we note any patterns. We can also look at race, gender, and see if some teachers are teaching better to a certain sub-population.

Key Questions We need to ask some key questions to find out why some students had better growth than others. We could even look at each subgroup individually and think about what contributes to negative and positive growth in the classroom.

Student Pattern Report – Key Questions Different experience? Different strategies? Different needs? Number of hours? In this case we are comparing students in the same sub-group, the group that is considered the H group. Some key questions we might want to ask include: After asking these questions we find that in this particular report the hours that a student participated in a program made positive difference in growth. At this point we looked at number of hours so we ran another report of all 31 students to see if it had a large effect on student growth. We looked at all students that had over 40 hours of enrichment/remediation etc…

Student Pattern Report – Key Questions Different experience? Different strategies? Different needs? Number of hours? YES! Rerun the report with new criteria.

Student Pattern Report – Next Steps All 31 Students in the Program 16 Students who attended for 40+ hours The 16 students that had over 40 hours in a program showed far greater growth than their counter parts that did not participate in the program. The 15 that didn’t participate the 40+ really negatively effected the overall growth. If you run a report and this is your result think the next step to figure out what the number actually mean. This shows the program did what you wanted.

Less Informed Conclusion: We need to change the selection criteria for this program. More Informed Conclusion: We need to adjust the recommended hours for participants.