1. Leverage ACT Aspire Results to Analyze Tasks, Student Work, and Examine Instruction
Spring 2018 Test Administration
Dr. Jorge Peña, Director of School Improvement and Accreditation

2. Prayer (Sign of the cross)
Loving God, Our creator, our savior, our companion, bless this journey of a new school year that we undertake. Refresh our souls and renew our spirits as we embrace the beautiful ministry you have called us to.
We welcome those who are new to our schools and ask that you strengthen them to share the wonderful gifts you have given them.
Lord, make our hearts pure as we prepare for the return of students to our schools, and may you guide them to return with open hearts and minds eager to learn.
We ask this through Christ our Lord.
Amen.
Prayer

3. Check-in Activity: Expectations
Find someone you do not know
Each person has one-minute to answer questions:
What is your name?
What is the name of your school?
What is your role?
What do you hope to get out of today’s workshop?
What summer activity would you like to share?
0:19
0:20
0:18
0:16
0:15
0:21
0:17
0:23
0:27
0:28
0:26
0:25
0:14
0:24
0:22
0:12
0:04
0:05
0:03
0:02
1:00
0:01
0:06
0:07
0:11
0:29
0:10
0:09
0:08
0:13
End
0:51
0:52
0:50
0:49
0:47
0:48
0:53
0:54
0:59
0:30
0:58
0:57
0:55
0:56
0:46
1:00
0:45
0:36
0:34
0:33
0:31
0:32
0:37
0:35
0:43
0:38
0:42
0:44
0:41
0:39
0:40
0:19
0:20
0:18
0:15
0:14
0:16
0:17
0:28
0:26
0:27
0:25
0:24
0:22
0:23
0:21
0:06
0:02
0:03
0:01
End
0:29
1:00
0:04
0:05
0:11
0:12
0:10
0:09
0:07
0:08
0:13
0:45
0:51
0:52
0:50
0:49
0:47
0:48
0:53
0:54
0:59
1:00
0:58
0:57
0:55
0:56
0:46
0:44
0:35
0:36
0:34
0:33
0:31
0:32
0:37
0:38
0:42
0:43
0:41
0:40
0:39
0:30

4. Workshop Norms Take an inquiry stance. Ground statements in evidence.
Assume positive intentions.
Stick to protocol.
Start and end on time.
Be here now.

5. Workshop Objectives
12: :55 Objective 1- Collaborate with partner to share data story
12:55 - 2:20 Objective 2- Identify the problem of learning by analyzing tasks and student work
2:20 – 2:30 Break
2:30 - 3:30 Objective 3- Discuss a process to examine instruction on the problem of learning

6. Archdiocese of Chicago Academic Improvement Goals
Reading and Math
Student are to learn the Archdiocese of Chicago Curricular Benchmarks in English/language arts and math. As measured by meeting or exceeding ACT Aspire College Readiness Benchmarks: “ready” or “exceeding”
Student growth from one year to the next is “high” or above the expected level of performance

7. Share analysis: what is the data story?
Collaborate with a partner from another school. Each of you has five minutes.
What do you notice about the spring 2018 results?
What wonderings do you have about the spring results?
5 minutes
5 minutes

8. Workshop Objectives
12: :55 Objective 1- Collaborate with partner to share data story
12:55 - 2:20 Objective 2- Identify the problem of learning by analyzing tasks and student work
2:20 – 2:30 Break
2:30 - 3:30 Objective 3- Discuss a process to examine instruction on the problem of learning

9. Improving Instruction and Student Learning
There are only three ways to improve student learning. The first is to increase the level of knowledge and skill the teacher brings to the instructional process. The second is to increase the level of complexity of the content that students are asked to learn. And the third is to change the role of the student in the instructional process. That’s it. If you are not doing these three things, you are not improving instruction and learning. Source: Instructional Rounds in Education (2009). Pg. 24
Table Talk: what resonates with you with this quote?

10. Instructional Task

11. Tasks predict student performance
Think of the task as the “ceiling” of what we would expect students to know
Tasks with low cognitive demands generate low cognitive student responses
Tasks with high cognitive demands generate high cognitive student responses
Task are high leverage because they predict student performance
University of Chicago’s Consortium on School Research

12. Harvard Graduate School of Education researchers analyzed tasks for each grade level
X-axis is the grade level, 0 is Kindergarten
Y-axis is the average assignment rating for the tasks assigned to student

13. We expect students to be assigned tasks that match the grade level, in other words grade 5 students are assigned tasks for grade 5

14. The orange line shows ratings for the task
The orange line shows ratings for the task. In grade 5, the average task was rated a 4.34 which is grade 4, in the third month
What do you notice about the average assignment rating for the grade levels?

15. I notice that primary grade tasks match the grade level

16. I also notice gaps in grade level tasks ratings emerge in grade 4 through grade 8. As we know students are taught how to read in grades K through 3, then instruction shifts in grade 4 to read to learn.

17. I also notice when students enter high school, the grade level task rating increases by a year and half

18. Consider Levels of Complexity: Webb’s Depth of Knowledge

19. What is learning? When does learning happen?
Remembering
Understanding

20. Questions to answer when analyzing a task
Analyze a task

21. Questions to answer when analyzing a task
What is the actual work students are being asked to do?
What do you have to know in order to engage the task?
What is the actual product of the task?
What is the task’s Level of Depth of Knowledge?
What is the distribution of performance among students in the class on the task?
If you were a student and did the task, what would you know and be able to do?
Grade-level meetings or faculty meetings. Analyze tasks on the skills on the CSIP.

22. Answer questions in silence
What is the actual work students are being asked to do?
What do you have to know in order to engage the task?
What is the actual product of the task?
What is the task’s Level of Depth of Knowledge?
What is the distribution of performance among students in the class on the task?
If you were a student and did the task, what would you know and be able to do?
10 minutes
Grade-level meetings or faculty meetings. Analyze tasks on the skills on the CSIP.
End

23. Group discussion- share answers
What is the actual work students are being asked to do?
What do you have to know in order to engage the task?
What is the actual product of the task?
What is the task’s Level of Depth of Knowledge?
What is the distribution of performance among students in the class on the task?
If you were a student and did the task, what would you know and be able to do?
5 minutes
Grade-level meetings or faculty meetings. Analyze tasks on the skills on the CSIP.

24. Rate the quality of authentic intellectual student work
Analyze student work samples

25. Gather data/evidence
Identify a skill that was challenging to students on the ACT Aspire
Create tasks that requires using the skill
Collect samples of student work on the task

26. Quality of Authentic Intellectual Work: Tasks
Standard 1: Knowledge Construction: The task calls for interpretation, analysis, synthesis, or evaluation of information.
3 = The task’s dominant expectation is for students to interpret, analyze, synthesize, or evaluate information, rather than merely to reproduce information.
2 = There is some expectation for students to interpret, analyze, synthesize, or evaluate information, rather than merely to reproduce information.
1 = There is no or virtually no expectation for students to interpret, analyze, synthesize, or evaluate information. The dominant expectation is that students will merely reproduce information gained by reading, listening, or observing.

27. Quality of Authentic Intellectual Work: Tasks
Standard 2: Elaborate Written Communication: The task asks students to draw conclusions or make generalizations or arguments and support them through extended writing. 4 = Explicit call for generalization and examples. The task asks students, using narrative or expository writing, to draw conclusions or to make generalizations or arguments, AND substantiate them with illustrations, details, or reasons. 3 = Call for generalization or examples. The task asks students, using narrative or expository writing, either to draw conclusions or make generalizations or arguments, OR to offer illustrations, details, or reasons, but not both. 2 = Short-answer exercises. The task or its parts can be answered with only one or two sentences, clauses, or phrasal fragments that complete a thought. 1 = Fill-in-the-blank or multiple-choice exercises.

28. Rate the writing samples of student work in the amount of time
5 minutes
End

29. Identify a Problem of Learning
Analyze student work

30. Identify a problem of learning
The problem is about “learning”
The learners themselves are not the problem
Learning = understanding + remembering

31. Make specific and objective observations from student work
(2 min) Silently examine the student work and write at least two “I notice” statements.
(5 min) Share & tune your “notice” statements with a partner. Strong “notice” statements are low in inference and are indisputable facts.
Example A, high inference statement: This student has not been taught how to write topic sentences.
Example B, low inference statement: Paragraphs in this piece of writing are not organized around one idea. For example, in paragraph 2, the student has sentences about monkeys, birds, and alligators, but the facts are not clearly connected to one another.
(3 min) Share with group
Time: 10 mins
Ahead of time:
Select and copy one piece of student work, making sure that it can be reviewed in just a couple minutes
Adjust time and directions as appropriate for the student work you have selected
Facilitation notes:
Review directions on slide
Possible adjustments:
Complete this as a whole group activity rather than in pairs if your group is small

32. What do you notice? What do you wonder?
(5 min) Silently examine all of the student work and take notes on what you notice and what you wonder
(5 min) Go around: What did you notice about the student work?
(5 min) Go around: What did you wonder about the student work?
(5 min) Open discussion: What patterns do we see that might shed light on why students are challenged with this skill? (5 min) Determine what additional pieces of student work we need to examine in order to identify the problem of learning.
Time: 40 mins
Ahead of time:
Write your school’s priority question in highlighted area
Adjust time and directions as appropriate for the student work you have selected
Facilitation notes:
Review the pieces of work the team has selected before beginning protocol
Review directions on slide
Answer any clarifying questions the team may have
Time each section to keep the team on pace
Possible adjustments:
Depending on quantity of student data, each faculty may look at a different examples of student work. For example, if you have 45 responses to a writing prompt, each person may examine 5 pieces of work.
You may also ask faculty to review student work or interim assessment data as pre-work so they are briefly reviewing work in the meeting they have carefully looked at beforehand
Depending on size of group, you may break the faculty into smaller work groups
Use an alternative looking at student work protocol that the team is familiar with

33. Identify a problem of learning
(5 min) Silently generate a possible problem of learning and write it on an index card
(5 min) Go around the group and read problems of learning
(5 min) As a group, come to a consensus on a problem of learning
Keep in mind: We’re looking for “a” problem of learning, not “the” problem of learning.
Time: 20 mins
Ahead of time:
Write your school’s priority question in highlighted text
Adjust time and directions as appropriate for the student work you have selected
Facilitation notes:
Review directions on slide
Remind faculty that we are not looking for THE perfect learner-centered problem. We are looking for A learner-centered problem worthy of further inquiry.
Possible adjustments:
Depending on size of group, you may break the faculty into smaller work groups. If you have multiple groups, each group will come up with one learner-centered problem and the room will come to consensus on the next slide. If you work as one group, add more time here and skip next slide.

34. Our Problem of Learning
We identified the following problem of learning:
[insert learner-centered problem here]

35. Debrief with SUMI Protocol
What Surprised you about analyzing tasks and student work?
How are you Understanding this approach to analyzing tasks and student work?
What is Marinating for you?
What Impact does analyzing tasks and student work have on you as leader of learning?
5 minutes
End

36. Break : 10 minutes 2:20 - 2:30

37. Workshop Objectives
12: :55 Objective 1- Collaborate with partner to share data story
12:55 - 2:20 Objective 2- Identify the problem of learning by analyzing tasks and student work
2:20 – 2:30 Break
2:30 - 3:30 Objective 3- Discuss a process to examine instruction on the problem of learning

39. Open Discussion
Why is observing practice challenging? What fears or concerns does it raise for us?
Time: 3 mins
Ahead of time:
Nothing
Facilitation notes:
Invite faculty members to share their own fears or predicted challenges around observing one another’s instruction and capture them on chart paper or on the presentation slide
Challenges could include teaching is very personal, getting feedback can feel stressful, observation is typically done for evaluative purposes, etc.
Possible adjustments:
Add a few minutes for silent write time before having people share out

40. The Purpose of Observing Instruction
IS to gather evidence that will help us understand how what we are doing (or not doing) as educators may be contributing to the problem of learning;
IS NOT to make judgments that will be used for external accountability.
Time: 2 mins
Ahead of time:
Nothing.
Facilitation notes:
Emphasize how the observations done as part of the Data Wise Improvement Process are to help faculty members learn and will not be used for external accountability purposes
Invite faculty members to share their own fears or predicted challenges around observing one another’s instruction
Possible adjustments:
Adjust the possible challenges based on your context
For detailed guidance about how to observe practice, see Key Elements of Observing Practice: A Data Wise DVD and Facilitator’s Guide linked in the Resources tab of this course

41. Leaning on Norms Take an inquiry stance Ground Statements in Evidence
Strategies for following this norm:
Use specific and descriptive language
Resist the urge to praise
Depersonalize by using “teacher” and “student” (instead of proper names) when taking notes
Take an inquiry stance
Ground Statements in Evidence
Assume positive intentions
Stick to protocol
Start and end on time
Be here now
Time: 5 mins
Ahead of time:
None.
Facilitation notes:
Explain that the Data Wise norms play an important role in Step 5. In particular,
Take an inquiry stance: as you observe, try not to judge but to understand what is happening in the classroom
Ground statements in evidence:
Use specific and descriptive language when observing practice. Building this skill is surprisingly difficult when discussing instructional practice, since people tend to have strong beliefs about good versus bad practices. (Will be discussed further later in this meeting)
Resist the urge to praise one another after observing practice. Even well-intended compliments can make an observation feel evaluative. Instead, teachers can simply express gratitude for sharing their practice
Depersonalize all instructional observations. This might include having faculty write”T” for “teacher and “S” for “student” to avoid using proper names
Assume positive intentions: Start with the belief that the teacher observed is trying to do right by students. Comment on what you see and hear, not on supposed intentions (which are hard to know)
Invite faculty members to discuss what it will look like and feel like when these norms are being followed when observing practice
Possible adjustments:
If you have adapted the Data Wise norms, feel free to use that list
If you have your own norms for observing instruction in a non-evaluative way, feel free to include those here

42. Leaning on Instructional Rounds Norms
Use the descriptive and specific voice
Separate the person from the practice
Honor differences and accept where people are at
Reward openness
Focus on building the next level of work not judging the current level of work
Everyone can improve
What do you notice?
What do you wonder?
Time: 5 mins
Ahead of time:
None.
Facilitation notes:
Explain that the Data Wise norms play an important role in Step 5. In particular,
Take an inquiry stance: as you observe, try not to judge but to understand what is happening in the classroom
Ground statements in evidence:
Use specific and descriptive language when observing practice. Building this skill is surprisingly difficult when discussing instructional practice, since people tend to have strong beliefs about good versus bad practices. (Will be discussed further later in this meeting)
Resist the urge to praise one another after observing practice. Even well-intended compliments can make an observation feel evaluative. Instead, teachers can simply express gratitude for sharing their practice
Depersonalize all instructional observations. This might include having faculty write”T” for “teacher and “S” for “student” to avoid using proper names
Assume positive intentions: Start with the belief that the teacher observed is trying to do right by students. Comment on what you see and hear, not on supposed intentions (which are hard to know)
Invite faculty members to discuss what it will look like and feel like when these norms are being followed when observing practice
Possible adjustments:
If you have adapted the Data Wise norms, feel free to use that list
If you have your own norms for observing instruction in a non-evaluative way, feel free to include those here

43. When Observing Practice
Observe lessons on the problem of learning
Try to stay for the full lesson
Focus on actions that are related to the problem of learning
Pay attention to what students are doing and saying (not just to what the teacher is doing and saying)
Feel free to interact with students about their work
Take detailed notes on student and teacher actions
Strive to have your notes specific and descriptive.
Time: 5 mins
Ahead of time:
Be aware that the next slide provides an example of a note-taking template, and the slide after that illustrates what specific and descriptive note-taking looks like
Facilitation notes:
Explain that there will be a lot of things to pay attention to in the classroom, so it is important to focus on actions that may shed light on why the learner-centered problem is what it is
Point out that it is easy to write only about teacher actions, but it is important to consider what teachers are asking students to do, and what students are actually doing
Let people know that, as long as it is not disruptive to the lesson, they can ask students about the work to help shed light on what students are and are not understanding in a lesson
Explain that you will now turn to discussing what it means to take specific and descriptive notes
Possible adjustments:
Feel free to modify this list based on your context or successful practices in the past

44. Review Observation Template
Time: 2 min
Ahead of time:
It might be helpful to fill in an example template so that teachers have an idea of what the notes could look like
Facilitation notes:
Introduce that this is a note-taking template you will use to help structure your notes, and keep them focused on the practice related to the learner-centered problem
Possible adjustments:
If you wish to make changes to the note-taking template, make sure to copy an image of your revised template so teachers are not confused.

45. Instructional Rounds Process to Observe Instruction
Get Oriented to the classroom (1 minute)
What grade is it?
What content area?
How many girls? How many boys?
How many adults are there?
How many minutes into the class are we?

46. Instructional Rounds Process to Observe Instruction
Examine the task
What are students being asked to do?
What are they actually doing?
Look for patterns of interactions
Are the interactions teacher-student-teacher?
Do students talk with each other?
Do students initiate conversation? Or are students always responding to the teacher?

47. Instructional Rounds Process to Observe Instruction
Listen to questions
What questions are being asked?
Who’s asking them?
What are the responses to the questions?
Consider time
How much time is spent on what activity?
Note time periodically throughout the observation as part of mapping what we see.

48. Instructional Rounds Process to Observe Instruction
Define evidence: descriptive statements of what is seen when examining instruction.
Not all forms of evidence are equally valuable
Some kinds of evidence is more helpful than others
Avoid “large-grained” evidence. For example, students are working in groups.
Use “fine-grained” evidence. For example, students worked individually even though they were in groups. Each worked on own paper and didn’t talk with others.
Evidence is only helpful if it helps the examiner understand something about the Instructional Core in that classroom. The Instructional Core is:
The teacher’s instructional process
Student engagement in the instructional process
The level and complexity of the content students are asked to learn

49. Objectivity/Specificity Matrix
Time: 5 min
Ahead of time: Nothing
Facilitation notes:
Introduce this matrix as a helpful way to practice taking specific and descriptive notes
Briefly define and give an example of each quadrant, using the following example to show how the same idea can be captured in different ways that are more or less helpful.
General judgmental: “That lesson was great.”
General and descriptive: “The teacher asked a lot of questions.”
Specific and judgmental: “The teacher asked a great question about why support for the treaty was mixed”
Specific and descriptive: “The teacher asked an open-ended question and let students think for 30 seconds before cold-calling on one.”
Note that we want to take notes that fall in the specific and descriptive box. Emphasize how specific and descriptive notes are more useful in gathering information to generate a problem of practice, and are also less likely to make this process unwelcoming for teachers.
Possible adjustments:
Feel free to adjust the example ahead of time for your specific context.

50. Objectivity/Specificity Matrix: How would you classify each observation? Work with a partner
The teacher missed an important opportunity to engage a student in discussing the mood of the poem
The teacher doesn’t know how to engage students in discussion
When the teacher asked “What are we hearing a lot of?” he answered the question and counted the number of times “rage” appeared in the poem
The teacher answered his own questions
Time: 5 min
Ahead of time: Nothing
Facilitation notes:
Ask faculty, individually, and then to share in pairs, to classify each of the 4 statements into the matrix
Have faculty share out where they placed each statement, guide teachers to correct answer
Possible adjustments:
If you are in a higher ed context, feel free to use the following slides 21 and 22 as alternatives

51. Objectivity/Specificity Matrix
Time: 3 min
Ahead of time: Nothing
Facilitation notes:
Review answers
Pose the question to teachers: What might happen if we use other judgemental or general statements to try to arrive at a problem of practice?
Possible adjustments:
If you are in a higher ed context, feel free to use the following two slides as alternatives

52. Practice Observing Instruction
Time: 5 mins
Ahead of time:
Have a member of the course team videotape a lesson that is related to the learner-centered problem, link to a relevant five minute clip from that lesson
Make note-taking templates for teachers, inserting the learner-centered problem
Facilitation notes:
Emphasize how, to build a culture where it is okay to observe one another’s classroom, that the course team member is setting an example by exposing themselves to this process first.
Remind teachers of the norms and matrix when writing observations.
Possible adjustments:
If you cannot videotape a lesson by this time, consider using the clip from the Step 5 skill building on the Introduction to Data Wise course
Use the Instructional Rounds
Observing Instruction Template

53. Practice Observing Instruction Use the Observing Instruction Template

54. Grade 5 Standards and Skills

55. Grade 5 Standards and Skills

56. Discuss
In pairs:
What are some specific, descriptive observations you made? What are some times you were judgmental or vague, and how could you revise those statements?
What strategies did you use to take notes?
What differences emerged between you and your partner?
Share with Group
Time: 10 min
Ahead of time: Nothing
Facilitation notes:
In partners (5 min): Have faculty review their observations and try to generate examples that are specific/descriptive, discuss strategies for taking notes, and major differences in notes that were taken
Share out (5 min): Have teachers share examples of specific and objective statements. If faculty are vague or judgemental, ask them clarifying questions to bring them back down (ex: “What do you see in the student work that makes you say that?”)

57. Affinity Protocol Individually (5 minutes)
Select 4 or 5 pieces of data that are specific and low-inference; and write each one on its own sticky note.
Break into groups of four.
In Groups (10 minutes)
First person reads one note out loud and places it on the chart
Others with similar notes read them out loud and place them near the first note
Repeat with a new person reading a note until all notes have been read and placed
Time: 15 min
Ahead of time:
Decide how to assign people to groups and include instructions for that on the slide. Consider whether it would be more useful for people to be with others who teach similar content and/or age groups
Make sure to bring sticky notes (5 notes per faculty member), markers and chart paper to the meeting
Facilitation notes:
Introduce this protocol as a way of analyzing the observations, and collectively coming to consensus on a common problem of practice
Read through the directions, taking questions
Possible adjustments:
It may be helpful to play the clip from the mini-lesson for the staff so that they have an example of how to approach this task. Directions detailed on earlier ‘optional’ slide.

58. Noticing Patterns In groups:
Sort the notes into categories that are relevant to the learner-centered problem
Give each category a name
Review the groupings and look for patterns
Write a phrase that captures each pattern
Time: 10 minutes
Time: 5 min
Ahead of time: Nothing
Facilitation notes:
Remind groups when it is time to switch into this round of the affinity protocol.
Briefly review instructions
Remind groups that there is no one right way of making these categories. All that matters is theses categories help you see underlying patterns about instruction that relates to the learner-centered problem.

59. Debrief with SUMI Protocol
What Surprised you about observing instruction or using the Affinity Protocol?
How are you Understanding this approach to observe and analyze instruction?
What is Marinating for you?
What Impact does observing instruction and analyzing observation have on you as leader of learning?
5 minutes
End

60. Review Next Steps Analyze ACT Aspire results with faculty
Create opportunities for faculty to collaborate to analyze tasks and observe instruction
View CSIP webinar: live webinar on 3:30 pm and will be recorded for on-demand viewing
Take an inquiry stance: or call questions to Dr. Jorge Peña:

62. + Plus ∆ Delta + Plus: what worked well about today’s class
∆ Delta: what to change for the next class