Interpreting and Using the ACT Aspire Interim Results Dr. Jorge Peña, Director of School Improvement and Accreditation Archdiocese of Chicago November 2017

Workshop Objectives Objective 1- Build assessment literacy: understand the skills the Interim assessments measure Objective 2- Build assessment literacy: understand how results are reported Objective 3- Use results to make adjustments to instruction

ACT Aspire Periodic Technical Manual Build Assessment Literacy: Understand the Skills the Assessment Measures Scavenger Hunt ACT Aspire Periodic Technical Manual

Debrief Scavenger Hunt What skills do the Interim tests measure? What did we learn about the complexity and rigor of the tasks the Interim tests measure?

What skills do the Interim tests measure? Reading: Key Ideas and Details, Craft and Structure – Inference and Knowledge Integration Math: Geometry, Number and Operations in Base 10, Operations and Algebraic Thinking, Grade Level Progress (17-20), Number and Operations – Fractions, Modeling, Integrating Essential Skills, Measurement and Data and FOUNDATIONS – questions from the year before (8-10) Science: Interpretation of Data, Evaluation of Models, Inferences, and Experimental Results, Scientific Investigation

What did we learn about the complexity and rigor of the tasks the Interim tests measure? DOK (Depth of Knowledge) DOK 1 – Low on Blooms Taxonomy (Remembering and Understanding) DOK 2 – Mid Blooms Taxonomy (Applying and Analyzing) DOK 3 – High Blooms Taxonomy (Evaluation and Creation)

Depth of Knowledge: Reading Interim DOK 1 – 20-35% (at most 4 questions) DOK 2 – 40-70% DOK 3 – 40-70%

Depth of Knowledge: Math Interim DOK 1 – 12-21% DOK 2 – 44-51% DOK 3 – 32-40%

Depth of Knowledge: Science Interim DOK 1 – 5-20% DOK 2 – 43-62% DOK 3 – 20-33%

Workshop Objectives Objective 1- Build assessment literacy: understand the skills the Interim assessments measure Objective 2- Build assessment literacy: understand how results are reported Objective 3- Use results to make adjustments to instruction

Ladder of Inference I observe objectively Helps stay focused on evidence

Ladder of Inference I select data from what I observe & I add meaning to what I have selected I observe objectively Helps stay focused on evidence

I make assumptions based on the meaning I have added Ladder of Inference I make assumptions based on the meaning I have added I select data from what I observe & I add meaning to what I have selected I observe objectively Helps stay focused on evidence

I draw conclusions which prompt feelings Ladder of Inference I draw conclusions which prompt feelings I make assumptions based on the meaning I have added I select data from what I observe & I add meaning to what I have selected I observe objectively Helps stay focused on evidence

I take action based on my beliefs and feelings Ladder of Inference I take action based on my beliefs and feelings I draw conclusions which prompt feelings I make assumptions based on the meaning I have added I select data from what I observe & I add meaning to what I have selected I observe objectively Helps stay focused on evidence

Ladder of Inference Climbing too quickly up the ladder I Select: some data - ACT Aspire Interim results - Conversations with teachers

Ladder of Inference Climbing too quickly up the ladder I add: interpretation - Students are doing poorly in reading I Select: some data - Quarterly reading benchmarks - Conversations with teachers

Ladder of Inference Climbing too quickly up the ladder I draw: conclusions - Curriculum is not adequate I add: interpretation - Students are doing poorly in reading I Select: some data - Quarterly reading benchmarks - Conversations with teachers

Ladder of Inference Climbing too quickly up the ladder I take: actions -We need a new reading curriculum I draw: conclusions - Curriculum is not adequate I add: interpretation - Students are doing poorly in reading I Select: some data - Quarterly reading benchmarks - Conversations with teachers

Ladder of Inference Disciplined Approach I Select: some data - ACT Aspire Interim results - Quizlets - Teachers’ notes from guided reading groups

Ladder of Inference I add: interpretation Disciplined Approach I add: interpretation - Analysis of Quizlets and Interim results indicate high students errors are due to guessing words I Select: some data - ACT Aspire Interim results - Quizlets - Teachers’ notes from guided reading groups

Ladder of Inference I draw: conclusions Disciplined Approach I draw: conclusions - Students are not regularly applying word-level decoding skills I add: interpretation - Analysis of Quizlets and Interim results indicate high students errors are due to guessing words I Select: some data - ACT Aspire Interim results - Quizlets - Teachers’ notes from guided reading groups

Ladder of Inference I take: actions Disciplined Approach I take: actions - Several possible actions (analyze the complexity of tasks, re-teach skill), but should be tied specifically to the multiple sources of data. I draw: conclusions - Students are not regularly applying word-level decoding skills I add: interpretation - Analysis of Quizlets and Interim results indicate high students errors are due to guessing words I Select: some data - ACT Aspire Interim results - Quizlets - Teachers’ notes from guided reading groups

Report: Subject Proficiency by Group 30,000 feet altitude view

Understand the Interim to Summative Concordance Tables

Report: Subject Proficiency by Group 30,000 feet altitude view

Report: Subject Proficiency by Group 30,000 feet altitude view

Report: Subject Proficiency by Student 15,000 feet altitude view

Report: Student Performance 10,000 feet altitude view

Report: Skills Proficiency by Student 5,000 feet altitude view

Report: Response/Content Analysis Plane has landed

Report: Response/Content Analysis

Report: Response/Content Analysis

Ensure you are at the “school” level www.ocs.archchicago.org

Interim Reports- Educator

Workshop Objectives Objective 1- Build assessment literacy: understand the skills the Interim assessments measure Objective 2- Build assessment literacy: understand how results are reported Objective 3- Use results to make adjustments to instruction

Adjusting Instruction and Improving Student Learning There are only three ways to improve student learning. The first is to increase the level of knowledge and skill the teacher brings to the instructional process. The second is to increase the level of complexity of the content that students are asked to learn. And the third is to change the role of the student in the instructional process. That’s it. If you are not doing these three things, you are not improving instruction and learning. Source: Instructional Rounds in Education (2009). Pg. 24 Table Talk: what resonates with you with this quote?

The teacher and the student in the face of the content. The relationship between the teacher, the student, and the content- not the qualities of any one of them by themselves. Increases in student learning occur only as a consequence of improvements in the level of content, teachers’ knowledge and skill, and student engagement. If you change any single element of the instructional, you have to change the other two.

Tasks predict student performance Think of the task as the “ceiling” of what we would expect students to know Tasks with low cognitive demands generate low cognitive student responses Tasks with high cognitive demands generate high cognitive student responses Task are high leverage because they predict student performance University of Chicago’s Consortium on School Research

Harvard Graduate School of Education researchers analyzed tasks for each grade level X-axis is the grade level, 0 is Kindergarten Y-axis is the average assignment rating for the tasks assigned to student

We expect students to be assigned tasks that match the grade level, in other words grade 5 students are assigned tasks for grade 5

The orange line shows ratings for the task The orange line shows ratings for the task. In grade 5, the average task was rated a 4.34 which is grade 4, in the third month What do you notice about the average assignment rating for the grade levels?

I notice that primary grade tasks match the grade level

I also notice gaps in grade level tasks ratings emerge in grade 4 through grade 8. As we know students are taught how to read in grades K through 3, then instruction shifts in grade 4 to read to learn.

I also notice when students enter high school, the grade level task rating increases by a year and half

Consider Levels of Complexity: Higher Order Thinking Skills

Consider Levels of Complexity: Bloom’s Taxonomy

Consider Levels of Complexity: Webb’s Depth of Knowledge

Whole-group or Small-group instruction: What percent of students should demonstrate proficiency on a skill in order to determine continuing whole-group instruction or providing small-group instruction? The optimal percent of proficiency to determine adjusting to small-group instruction is 66%.

What is the success rate? The optimal success rate for fostering student achievement is 80%. A success rate of 80% shows that students are learning the material, and it also shows that students are challenged.

Questions to answer when analyzing a task What is the actual work that students are being asked to do? What do you have to know in order to engage the task? What is the actual product of the task? What is the distribution of performance among students in the class on the task? If you were a student and did the task, what would you know and be able to do? Grade-level meetings or faculty meetings. Analyze tasks on the skills on the CSIP.

Questions to answer when analyzing a task What is the actual work that students are being asked to do? What do you have to know in order to engage the task? What is the actual product of the task? What is the distribution of performance among students in the class on the task? If you were a student and did the task, what would you know and be able to do?

Analyze an ACT Aspire Interim test item Use the Response/Content Analysis Report Identify a skill with a proficiency rate less than 66% Analyze a test item for that skill by answering the five questions: What is the actual work that students are being asked to do? What do you have to know in order to engage the task? What is the actual product of the task? What is the distribution of performance among students in the class on the task? If you were a student and did the task, what would you know and be able to do?

Review Next Steps: Bring training back to the faculty at school Build assessment literacy- identify the skills the ACT Aspire Interim assessments measure Build assessment literacy- understand how results are reported Collaborate with teachers to analyze ACT Aspire Interim results Compare/contrast the complexity of tasks between the ACT Aspire Interim test items and student tasks assigned by teachers Identify the skills that require small-group instruction for re- teaching Re-teach the skills that have a proficiency rate less than 66% Document and showcase the improvement journey: meeting agendas with notes, pictures, videos Email questions to Dr. Jorge Peña, jpena@archchicago.org