1. October 24-25, 2011 - Chapel Hill, NC
Measures of Student Learning Design Groups World Language Essential Standards
October 24-25, Chapel Hill, NC
Ann Marie Gunter, World Language Consultant

2. Our Purpose:. To gather structured feedback from
Our Purpose: To gather structured feedback from design group members on what meaningful assessment looks like for the content area of World Languages
Outputs
The Rater Sheet
Drivers & Restrainers Chart
Annotation of Facilitator’s Copy of SCoS
Support for Item Writers Questions

3. Day 2 Agenda Tuesday, October 25 8:00 – 10:00 a.m. Round One Ratings
10:00 – 12:00 p.m. Group Discussion of Round One Ratings
12:00 – 1:00 p.m. Lunch
1:00 – 2:00 p.m. Round Two Ratings
2:00 – 3:00 p.m. Round Three Ratings: Going Deeper/Annotating the SCoS
3:00 – 4:45 p.m. Support for Item Writers Activity
4:45 – 5:00 p.m. Wrap-Up

4. World Language Work Group Facilitators
ROLE
NAME
K-12 World Language Lead Facilitator
Ann Marie Gunter
Classical Languages Support Facilitator
Sherry Broome
Dual & Heritage Languages Support Facilitator
Fran Harris-Burke
Modern Languages Support Facilitators
Novice Low – High Proficiency
Intermediate Low – High Proficiency
Advanced Low – Mid Proficiency
Geetanjali Soni
Robin Loflin Smith
Julian Nichols Wilson
US Education Department Support Facilitators
Victoria Cirks
Lisa Shimmel

5. Housekeeping Breaks as needed Ground rules Parking Lot Lunch (12:00)
Please note there are no specifically scheduled breaks during these days. Please feel free to break whenever you feel it is necessary!

6. WLES Wiki http://wlnces.ncdpi.wikispaces.net/ Content Sessions-MSL
Go online and show the different parts/pages of the wiki

7. Round One Ratings (~8-10 a.m.)
Task 1: Finish any standards that have not been completed with Cognitive Rigor (Questions 1 and 2)
Task 2: Rate the following -
Criticality of Standards (Questions 3 and 4)
Most Appropriate Assessment Items (Questions 5 and 6)
Reference: MSL Item Specification Questionnaire Key
Please work individually
After the support facilitator welcomes the small design groups back, the group will move into the completion of the Round One part of the rater sheet. Here, all groups will be providing their individual feedback on the criticality of the standards and what type of items will best assess them. These questions (questions 3 – 6 for courses with Essential Standards
For Round One, the teachers should complete these individually. Of course,
they don’t need to sit in complete silence! The goal here, however, is to record their
individual thoughts.
Role of the Facilitator: Circulate throughout the small design group, answering questions or providing support as needed.
Note: the next 3 slides provide a visual of the 6 questions and possible responses that will be rated, which can be referenced, as needed. Use the handout “MSL Item Specification Questionnaire Key” for assistance with completing this task.

8. Revised Bloom’s Taxonomy
The Knowledge Dimension
The Cognitive Process Dimension 1
Remember 2
Understand 3
Apply 4
Analyze 5
Evaluate 6
Create
Factual
Knowledge
Conceptual   
Procedural
Metacognitive

9. Question 1 (Q1) Rating Activity
Rate the Cognitive Rigor of the Clarifying Objectives from the World Language Essential Standards according to RBT:
Q1: Cognitive Process (Cog.)
Use the alphabetized RBT reference sheets
Work individually or with a partner in your group 1 2 3 4 5 6
Remember
Understand
Apply
Analyze
Evaluate
Create
Round One: Teachers working with Essential Standards will complete questions 1 and 2 for as many of the clarifying objectives as they can in the afternoon on Day One. Teachers working with the Common Core will not fill out anything on the rater sheet until the second day.

10. Round One Ratings (~8-10 a.m.)
Cognitive Rigor (continued from Day 1)
Rate the Cognitive Rigor of the Clarifying Objectives from the World Language Essential Standards according to RBT:
Q1: Cognitive Process (Cog.): Cognitive Process Dimension
Q2: Knowledge Type (Know.): Knowledge Dimension 1 2 3 4 5 6
Remember
Understand
Apply
Analyze
Evaluate
Create 1 2 3 4
Factual
Conceptual
Procedural
Meta-Cognitive

11. Round One Ratings (~8-10 a.m.)
Criticality
Q3: Importance (Imp.)
Q4: Time 1 2 3 4 5
Slightly important
Important
Very Important
Extremely Important
Critically Important 1 2 3 4 5
Significantly less time
Less time
Similar time
More time
Significantly more time

12. Round One Ratings (~8-10 a.m.)
Item Type
Q5: Best opportunity to demonstrate mastery (Best)
Q6: Fully assessed through selected response (SR) SR SA ER PT PF
Selected Response
Short answer
Extended Response
Performance Task
Portfolio 1 2 3
Cannot be assessed
Partially assessed
Fully assessed

13. Round One Ratings (~8-10 a.m.)
Task 1: Finish any standards that have not been completed with Cognitive Rigor (Questions 1 and 2)
Task 2: Rate the following -
Criticality of Standards (Questions 3 and 4)
Most Appropriate Assessment Items (Questions 5 and 6)
Reference: MSL Item Specification Questionnaire Key
Please work individually
After the support facilitator welcomes the small design groups back, the group will move into the completion of the Round One part of the rater sheet. Here, all groups will be providing their individual feedback on the criticality of the standards and what type of items will best assess them. These questions (questions 3 – 6 for courses with Essential Standards
For Round One, the teachers should complete these individually. Of course,
they don’t need to sit in complete silence! The goal here, however, is to record their
individual thoughts.
Role of the Facilitator: Circulate throughout the small design group, answering questions or providing support as needed.
Note: the next 3 slides provide a visual of the 6 questions and possible responses that will be rated, which can be referenced, as needed. Use the handout “MSL Item Specification Questionnaire Key” for assistance with completing this task.

14. Round One Discussion (~10-12)
Support Facilitators Lead Discussions in Working Groups
Participants share responses to Questions 3-6
Sharing multiple points of view, especially where there are differences, is encouraged!
Round One Discussion, 10:00 a.m. – 12:00 p.m.
During this time, design group members should share their answers to what they have
answered on the rater sheet. There are a few ways that the support facilitator could help with this process:
• Having them raise hands for what numbers or letters they coded in, and discussing
standards where two or more answers seem to be popular
• Doing a round-robin for teachers to share what they coded for particular standards.
• Any other ways that you normally ask teachers to share out in group sessions!
In the past, when this type of activity has been done in the development of EOGs and EOCs,
the level of agreements tends to be fairly high.
Role of the Facilitator: The support facilitator will lead the teachers in sharing and
discussing their answers to questions 3-6 (Essential Standards) or questions 1 – 4
(Common Core). Specific attention should be paid to areas where there are differences of opinion. Encourage the teachers to share their points of view!
NOTE: ask each facilitator to select a recorder to capture any salient points from group discussion

15. LUNCH
12noon – 1pm

16. Round Two Ratings (~ 1:00-2:00 p.m.)
Task: Complete column 2
Indicate / if answers have not changed from Round One
If answers have changed, code new responses
Please work individually
Round Two Ratings, 1:00 p.m. – 2:00 p.m.
Given the discussion that took place in the morning, some of the teachers may have
changed their minds about the criticality or best type of items to assess some standards.
During this hour, they will fill out the questions for column 2. They are answering the same questions as during Round One, just with new information and dialogue in mind. If their answers have not changed from Round One, they can place a / in the box. If their answers have changed, they can code in the new responses. Again, the idea is for us to get their individual feedback, but they can discuss as they complete Round Two.
Role of the Facilitator: Circulate as the teachers reconsider their answers.
Please note that this agenda is flexible! The priority is for the rater sheet to be
complete! If your group needs more time to complete the sheet, please allow that.

17. Round Three (~ 2:00-3:00 p.m.) Annotating the Standards
Task 1: ID standards marked “partially assessable” with Selected Response (Q6 – Answer 2)
* these standards on Facilitator’s Copy for each Work Group
Task 2: Consider standards one at a time
Identify which part of the standard IS ASSESSABLE with Selected Response
Underline this part of the standard in the Facilitator’s Copy for each Work Group
Round Three, 2:00 p.m. – 3:00 p.m.
The support facilitator should ask a participant or a United States Department of Education facilitator to serve as a recorder for this part of the day. The recorder should have the facilitator’s copy of the Common Core or Essential Standards that was provided in your plastic bin.
Output: Annotation of Facilitator’s Copy of Common Core or Essential Standards
The facilitator should ask the teachers to identify any standards they marked as “partially assessable” with a selected response item. For Common Core teachers, these are standards for which the teachers marked “2” for question 4 on their rater sheets. For Essential Standards teachers, these are standards for which the teachers marked “2” for question 6 on their rater sheets. The recorder should star these standards in the facilitator’s copy.
Next, consider each one of those standards one at a time. Ask the teachers to identify which part of the standard IS ASSESSABLE with a selected-response item. Have the recorder underline this part of the standard in the facilitator’s guide. Mark up each of the set of standards that your design group is considering in this way.
This type of output provides much greater specificity on what the teachers meant when
they indicated that a standard was “partially assessable” by a selected-response item.
Facilitator’s Role: Help the group to reach consensus on which standards are “partially
assessable.” Then, please lead them in a discussion and ask questions that help get them to
a point where they are articulating what part of the standard should be underlined by the
recorder. At 3:00 p.m., please collect the rater sheets and put them in the provided brown
envelope. The annotated standards should go in another brown envelope that will be
provided for you.

18. Support for Item Writers (~ 3:00-4:45 p.m.)
Task: Answer the questions in your folder
Looking for individual feedback
Can work together as appropriate to group
Purpose:
To provide feedback to the writers for items that will be vetted by teachers in the spring
To provide a narrative answer to the question,
“What does meaningful assessment look like in your content area?”
To capture more qualitative information that is not easily gathered by rater sheets and annotated standards.
Support for Item Writers Activity, 3:00 p.m. – 4:45 p.m.
In their folders, the teachers will have a set of questions that will provide feedback to the individuals that actually write the items that the teachers will vet in the spring. The
questions are fairly self-explanatory and provide space for free-response answers. Again, we are looking for individual feedback, but the teachers can work together as they would like.
Output: Support for Item Writers Questions
These questions provide the narrative component to the answer “what does meaningful assessment look like in your content area?” These questions will capture more qualitative information that is not easily gathered by the rater sheets and the annotated standards.
Role of Facilitators: The teachers should not need much assistance with this part of the day.
Please circulate and answer questions as needed! At 4:45 p.m., please collect the rater
sheets and put them in the provided brown envelope.

19. Curriculum & Instruction World Languages
Helga Fasciano
Section Chief of K-12 Program Areas
Ann Marie Gunter
World Language Consultant
NCDPI World Languages Website
WLES Wiki

20. Closure and Next Steps
Please complete online survey to be sent within the next week or so Look for communications from Jenn Preston re: future meetings
Reimbursement questions?
reference prior communications and/or
contact Jenn Preston at: (919) or
Wrap-Up, 4:45 p.m. – 5:00 p.m.
Please thank the teachers for their hard work! In the next week or so, they will receive an online survey to provide feedback on how the meeting went. They will also receive
communication from Jenn about meetings in the future.
Also, they may have questions about how reimbursement will work. This information was sent to them before the meetings, so please ask them to either ask me or reference the that they received with pre-work.
As the facilitators depart, please return the plastic bins, supplies, and any extra copies you may have – we will reuse them! You should have three brown envelopes to return to the facilitators’ table: one with rater sheets, one with annotated standards, and one with the support for item writers questions.
And again, THANK YOU for your incredibly hard work. This work could not be a success
without you.