1. Professional Learning: NYC Performance Task Norming Workshop
10/14/2015
Queens North Borough Field Support Center Lawrence Pendergast, Director
Queens North Borough Field Support Center Lawrence Pendergast, Director
Professional Learning:
NYC Performance Task Norming Workshop
October 14, 2015
Deirdre Keyes, Deputy Director for Teaching and Learning, Queens North BFSC
(1 minute) 1

2. Rate these newspapers on a scale of 0 – 4 (4 being the highest)
10/14/2015 OR
( 5 minutes ) Activity
Raise your fingers to indicate your rating.
Why are you having difficulty rating these papers? Do we have a consensus?
Possible answers: We all may have rated each newspaper differently based on how and why we read each newspaper. We might have different purposes as readers, based on what we’re reading. However, if I had provided you with a RUBRIC an evaluation tool / such as a set of guidelines I would have an expectation of consistent outcome/response. We would have “normed” our outcomes.
Today’s workshop focuses on Norming NYC Performance Tasks which are scored on a rubric. However, prior to any scoring activities, all teachers must be trained on how to consistently and accurately apply the rubric to student work. This process is know as “norming.” We will take you through this process today.

3. 10/14/2015
Guiding Questions
What steps are necessary to consistently and accurately norm the NYC Performance Tasks?
What tools are available to support this work?
( 1 minute )
*Note that while it is important to norm scoring of all assessments scored at school sites, this training is designed for NYC Performance Tasks norming and not for Running Records of Literacy, WebABLLS, or SANDI/FAST.”

4. Session Outcomes: Participants will be able to:
10/14/2015
Session Outcomes:
Participants will be able to:
Understand the design of the NYC Performance Tasks
Develop shared expectations for student work
Consistently apply the NYC Performance Task rubric
(1 min)
Understand the design of the NYC Performance Tasks and the importance of norming scoring practices.
Develop shared expectations for student work around each of the traits in the NYC Performance Task Rubric.
Be able to consistently apply the rubric when scoring student work.

5. NYC Performance Tasks: Norming Assessments Professional Learning Kit
10/14/2015
NYC Performance Tasks: Norming Assessments Professional Learning Kit
Part 1
Facilitator Note-Taking Template
Norming PL Kit – Part One
Activity 1 – Task Design & Requirements
Activity 2 – Using the Teacher Scoring Guide
Norming PL Kit – Handouts
Part 2
Facilitator Note-Taking Template
Norming PL Kit – Part Two
Activity 3 – Using Completed Student Tasks
Activity 4 – Trait Deep Dive
Norming PL Kit – Handouts
Additional Resources
School Leader Guide
Facilitator Guide
( 3 minutes )
The Norming Kit outlines the norming process schools should complete prior to scoring Beginning Of Year and End Of Year assessments.
Norming Kit is divided into 2 parts:
Part 1 is designed for schools and teacher groups who may be new to the norming process
Provides an overview of the performance tasks, familiarizes teachers with the task design and rubric, and asks teachers to score sample student work - comparing their own scores to those provided in the Teacher Scoring Guides.
Part 2 is designed for schools with extensive norming experience who have already mastered the content covered in Part 1 and are ready to norm their own completed student tasks using the rubric.
*Note: Norming Kits found on Advance Intranet

6. 10/14/2015
GROUP DISCUSSION: Why are normed scoring practices important for instruction?
(4 min) - GROUP DISCUSSION: Why are normed scoring practices important for instruction?
Allow 2 minutes for group members to share ideas and suggestions – Reveal the list and compare to group discussions.
Use the last 2 minutes to highlight any bulleted responses above which the group did not identify
Norming sessions set a precedent for coming together as teacher teams to review student work.
Norming sessions are an opportunity for collaborative teacher teams to engage in rich discussion of student work and student progress.
Norming helps teacher teams to clarify instructional objectives and build a team culture that is focused on student outcomes.
Norming builds teachers’ capacity to construct their own rubrics.
Norming saves time in the scoring process because teachers have a clear understanding of the evidence required for a student to meet expectations at each level of performance on each trait.
Norming ensures reliable results, which are the key to producing actionable student learning data.

7. NYC Performance Task Norming Activities Work Session – Part I
10/14/2015
NYC Performance Task Norming Activities Work Session – Part I

8. Rubric: Key Features 10/14/2015 (2 min)
“Take moment to look at the NYC Performance Task Rubric. What do you notice about how it is structured? What key features do you notice?”
Highlight anything about the structure and key features that participants do not note:
“Each performance task has a similar structure and several key features.”
“First, the left most column of the rubric identifies the key standards and traits on which student work will be assessed.”
“For each trait, the rubric describes expectations for student work at each level of performance, Level 4: Exceeding Expectations – Level O: No Evidence.”
“A score of zero should be considered carefully and only given to students who did not attempt the question or copied directly from the task or text. A score of 4 should be given to a student who consistently exceeds the standard and demonstrates knowledge or skills beyond what is required to complete the task.”
“In addition, students can receive half points on the 4 point scale.”
*Note: Schoolnet answer sheet is ordered Level 0 – Level 4, while the rubric is ordered Level 4-Level 0. Thus, teachers will need to be careful about transferring their scores from the rubric to the Schoolnet answer sheets.

9. Highlight the Rubric
10/14/2015

10. 10/14/2015
NYC Performance Tasks: Norming and Scoring Professional Learning Kit (Part One) Activity #1 – Task Design and Requirements
Objective
In grade/subject teams, participants will be able to interpret the NYC Performance Task by analyzing the requirements for students and predicting students’ strengths and struggles.
Activity
Preview the NYC Performance Task and consider the four key questions on Handout A.
Use the handout to take notes while you review the task and texts.
Review the rubric and note any similarities and/or differences to your own reflections.
Share your findings with your team.
(15 min) Distribute / Identify materials for participants:
Sample NYC Performance Task
Handout A (one copy/participant)
Rubric (not to be distributed until teachers have completed Handout A) (one copy/participant)
We’re going to take a few minutes to review a sample NYC Performance Task. As you’re reviewing the task, we’d like to pay attention to two key questions. First, what did students need to know in order to complete the assessment? Second, what did they have to do? After considering those two questions, imagine how students in your schools approached this task? Where were they probably able to show off their strengths? Where did they likely struggle?”
Ask participants to take 5 minutes to review the task and texts and taking notes on the Handout A.
Invite participants to share their responses and reflections for 5 minutes in pairs or aloud to the whole group.
Finally, hand out the actual rubric for the task and have participants compare their notes and reflections to those listed in the rubric. Where are there similarities? Where are there differences? Encourage them to share their reflections in pairs or aloud to the whole group (5 min)

11. Activity #1 – Task Design
10/14/2015
This slide is for illustrative purposes only – handout is provided, but may use when introducing activity to let participants know what handout they should be looking at.

12. 10/14/2015
NYC Performance Tasks: Norming and Scoring Professional Learning Kit (Part One) Activity #2 – Norming Scoring Practices Using the Teacher Scoring Guide
Objective
In grade/subject teams, participants will be able to apply the rubric as an evaluative tool for scoring sample student work and compare their scores against those provided in the Teacher Scoring Guide.
Activity
Preview the NYC Performance Task and the sample of student work provided on the redacted scoring guide.
Using the rubric, review and score the piece of student work – making notes where appropriate.
Conclude by comparing your results to the scored student work and completed scoring matrix from the Teacher Scoring Guide and discuss any discrepancies.
(20 min) Distribute/Identify Materials for Participants:
Rubric (participants should already have this from Activity #1)
Redacted sample of student work from the Teacher Scoring Guide (annotations and scores have been whited out)
Same (non-redacted) sample of student work with completed scoring matrix from the Scoring Guide with annotations. (NOTE: Please wait to distribute the scored sample and corresponding scoring matrix until after participants have completed their independent scoring).
Handout B (one for each participant; if reviewing a Math task provide two for each participant)
Introduce the Norming and Scoring Activity: “School teams will be responsible for norming their scoring practices by using the rubric. It is important that all scorers are calibrated to the rubric to ensure fair grading practice across students, teacher and schools. In this activity we will walk through the norming process using an example of student work from the Teacher Scoring Guide associated with your task.”
“For this activity you will receive two copies of the same student work sample – one with annotations and scores and one without. You will use the redacted version first to score the student work and then use the non-redacted version to compare your answers and justification to those in the Scoring Guide.”

13. Norming Steps for Activity #2
10/14/2015
Independently, score the sample student work provided (Sample A). You can use Handout B to capture your notes as you go (10 min)
Have the facilitator hand out the annotated version of the same sample of student work (Sample A) from the Teacher Scoring Guide (5 min)
Have the facilitator review the evidence for each trait and scores on the annotated sample student work (5 min)
Discuss any discrepancies between these scores and your own scores (15 min)
As displayed on the slide, there are four steps to this initial norming activity. Group norming activities such as this should begin by reviewing the rubric and discussing expected performance at each performance level for each rubric trait which we’ve already done in Activity 1
Activity 2 will start with independently scoring the sample student work provided. Feel free to either make annotations directly on the sample or on Handout B.”
“Once everyone has finished scoring, go around the table and reveal your scores trait by trait. If there seems to be a consensus on the trait score you can move onto the next one. If scores for a trait are more than 1 point apart, be sure to stop and have a quick discussion about that trait so that the group is able to reach a consensus.”
“Next, I (the facilitator) will hand out the annotated and scored sample of student work from the Scoring Guide for you to review in your table teams and compare to the scores your group decided on. As you compare scores, be sure to take note of any areas where you have questions or notice discrepancies.”
“Finally, we will discuss any outstanding questions or discrepancies you found between your group’s scores with those provided in the sample and try to reconcile them.”
“Please note that the sample student work and Scoring Guides we are using today are also available online through Schoolnet.”

14. Sharing Best Practices
10/14/2015
Sharing Best Practices
On an index card write down any successes you’ve had with Norming Assessments during this activity or at your school
Share your successes with the table
(5 minutes) - Handouts: Poster Paper / Markers
Ask participants to discuss in their tables and come up strategies they used during the 2 activities that made their norming sessions successful.
Make sure your observations are evidence-based: When discussing student work make sure your observations are grounded in the language of the rubric and based on evidence from the student work.
Be mindful of “airtime”— Make sure everyone has an opportunity to be heard and explain their scores using language from the rubric.
Discuss and reconcile: It is okay for participants to disagree about a score. This discussion and reconciliation allows for professional learning and helps to standardize expectations for student work.
Don’t lose sight of the big picture: Disagreement is good, but don’t get too far off topic. Remember where you want your group to be at the end of the session – a shared understanding of the evidence required for a student to meet expectations at each level of performance on each trait

15. NYC Performance Task Norming Overview of Part II Activities
10/14/2015
NYC Performance Task Norming Overview of Part II Activities

16. Independently score two (2) completed student tasks using the rubric.
10/14/2015
NYC Performance Tasks: Norming and Scoring Professional Learning Kit (Part Two) Activity #3 – Norming Scoring Practices Using Completed Student Tasks
Objective
In grade/subject teams, participants will be able to apply the rubric as an evaluative tool for scoring student work and establish a shared understanding of the expectations of the NYC Performance Tasks.
Activity
Independently score two (2) completed student tasks using the rubric.
Compare your scores for each trait as a group and reach a consensus on a score for each trait.
Repeat the process for the third completed task.
( 3 min) Distribute / Identify materials for participants:
Rubric (participants should already have this from Part 1: Activity 1)
Two (2) samples of completed student tasks from teachers
Handout B (2 copies for each participants; if scoring a Math task provide each participant with 4 copies)
Introduce the Norming and Scoring Activity: “School teams will be responsible for norming their scoring practices by using the rubric. It is important that all scorers are calibrated to the rubric to ensure fair grading practice across students, teacher and schools. In this activity we will walk through the norming process again as a group.”

17. 10/14/2015
NYC Performance Tasks: Norming and Scoring Professional Learning Kit (Part Two) Activity #4 - Rubric Trait Deep Dive
Objective
In grade/subject teams, participants will be able to apply the rubric as an evaluative tool for scoring student work and establish a shared understanding of the expectations of the NYC Performance Tasks.
Activity
Choose one trait from the rubric on which your team had difficulty reaching consensus in Activity #3.
Focusing on the trait you selected, review the student work samples and scoring matrices in the Scoring Guide.
Independently score two (2) completed student tasks, focusing on the trait you selected.
Discuss your scores with your table until you are able to reach consensus.
(3 min)
Distribute/Identify Materials for Participants:
Rubric (participants should already have this from Part 1: Activity #1)
Teacher Scoring Guide (including all student work samples, annotations and completed scoring matrices)
Two (2) samples of completed student tasks from teachers (ideally showcasing different ability levels)
Introduce the Norming and Scoring Activity: “In this activity, teachers will have an opportunity to closely examine one trait on the rubric on which their team struggled to reach consensus in Activity #3. Focusing on this one trait, participants will review completed scoring matrices from the Teacher Scoring Guide and score their own completed student tasks.”

18. NYC Performance Task Norming Resources
10/14/2015
NYC Performance Task Norming Resources

19. 10/14/2015
For any additional information or support please reach out to your Teacher Development Evaluation Coaches
COMMUNITY DISTRICTS: DISTRICT
Teresa Caccavale 24
Debbie White-Winkler 25
Brandon Contarsy 26
Erica Urena-Thus 30
HIGH SCHOOLS:
Lisa Grevenberg Districts 24, 25, 26
Tracy Atkins Renewal HS
Angelica Beissel New Visions
( 1 minutes)

20. Online Resources: Advance Intranet – Norming Kit
10/14/2015
Online Resources:
Advance Intranet – Norming Kit
School Net – NYC Performance Tasks
Assessment Administration Guide – NYC Performance Tasks - Guide
Wiki Space – Resource Links
( 2 minutes )

21. Participant Feedback Forms
10/14/2015
Participant Feedback Forms
Please complete the feedback form found in the back of your folder or online by going to
( 1 minute )