1. Smarter Balanced ELA Interim Assessment Scoring
Welcome….
Teacher Hand Scoring System (THSS) materials

2. Interims: Purpose and Intent
Support learning and teaching
Intended uses:
Check student progress
Inform classroom instruction
Practice with accessibility tools
Smarter Balanced designed the interim assessments for instructional use to support learning and teaching throughout the school year in order to
-allow teachers to check student progress
-provide information to improve classroom instruction
-provide practice with accessibility tools used on the summative
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

3. Benefits of Hand Scoring
Calibration:
Teachers calibrate scoring
Student responses calibrated to fixed rubric
Scoring aligned to Smarter Balanced assessments
Materials consistent across grade levels, content areas, item types
Interpretation of student learning:
Allows teachers to identify areas of strength and weakness
The greatest benefit of hand scoring is that it allows teachers to interpret strengths and weaknesses for their students in the particular skill being assessed. Student responses can be one piece of evidence (of multiple measures). Similar to formative assessment processes used in the classroom, teachers can evaluate the constructed responses to determine next instructional steps.
On a larger scale, these items allow for calibration to a fixed rubric, to provide a consistent ruler among students and teachers, and they help teachers and students better understand expectations of the summative assessment.
Across grade levels, content areas and item types, THSS materials are formatted in similar styles. Each item has a training guide and exemplars.
Washington State Office of Superintendent of Public Instruction
9/17/2018

4. Interim Assessment Process
Step 1: Planning and Preparation
Use Assessment Viewing Application (AVA), the interim blueprints (inside Test Administration Resources), and
THSS Materials (inside TIDE)
Step 2: Administration and Scoring
Use Test Administration interface (TA) to manage student tests
Train and score constructed response items in Interim Teaching Hand Scoring System (THSS)
The Teacher Hand Scoring materials are used in the midst of a set of assessment events.  
To help them determine which Interim assessment to give to students, educators should access the Assessment Viewing Application (AVA), the teacher hand-scoring (THSS) materials located in TIDE, and the interim blueprints posted in the General Information folder on the WCAP Portal.
The Test Administration interface and the Washington Secure browser are used to administer the Interim Comprehensive Assessments (ICA) and the Interim Assessment Blocks (IAB) to students.
Student responses to constructed response items are then scored at the local level via the Teacher Hand Scoring System by educators, using the THSS materials.
After hand-scoring is completed, data is available in both the Online Reporting System (ORS) and in the AIR Ways reporting system. If no hand-scoring is needed, which is true for many of the IABs, scores are available in ORS and AIR Ways soon after the completion of testing.
Each of these systems has their own User Guide and Training Module available on the WCAP Portal. Please refer to those materials for more specific questions about ORS and AIR Ways.  
Step 3: Reflection
View summary reports in the Online Reporting System and/or more detailed item-level and student-level reports in AIR Ways Reporting

5. Step 1: Planning and Preparation
Assessment viewing application (ava)
Interim blueprints
Teacher hand scoring system (thss) materials in tide

6. Planning: Using AVA Purpose: What is available to view:
Allows educators to view interim assessments for planning and preparation
Provides an understanding of content being assessed and time needed to complete interims
What is available to view:
Actual interim assessment items presented online in the same format as students will see the items
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

7. Planning: Available IABs
Hand Scoring
Grades 3–7
Grade 8
High School
Read Literary Texts
Read Informational Texts
Brief Writes
Revision*
Edit/Revise***
Language and Vocabulary Use*
Editing*
Listen/Interpret
Research**
Performance Task
*IAB is new for 2016–17
** IAB includes new items in 2016–17
*** The Edit/Revise IAB for grade 8 will be separated as soon as possible when items are available in the item bank 
This is a list of ELA Blocks for
The check marks indicate IABs that have some hand-scored items.
Most of these blocks contain 1-2 hand-scored items, with the exception of brief writes which are all hand-scored.
Specific writing types for Performance tasks can be seen in AVA.
G3: Opinion, G4-5: Narrative, G6: Argumentative, G7: Explanatory, G8: Explanatory, HS: Explanatory
For more information about the interims:

8. Planning: Using Interim Blueprints
Access blueprints: Test Coordinator Resources > General Information > Blueprint ELA/Literacy Interim Assessment
Sample:
The Blueprints show the break down for each block: total number of items, targets assessed with accompanying DOK, detailed information about the items, such as how many are machine scored and how many are short text (meaning they would need hand scoring). Note: Some are labeled as human scored rather than short text.
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

9. Planning: Time for Training and Scoring
9/17/2018
Planning: Time for Training and Scoring
Training:
Allow time for training PRIOR to scoring in the system
1–2 hours per training guide and exemplars set
Training time should decrease with familiarity
Scoring:
Time varies by item and number of student responses
Washington State Office of Superintendent of Public Instruction
9/17/2018
Washington State Office of Superintendent of Public Instruction

10. Preparation: Materials in TIDE
The complete set of THSS materials is located in TIDE. Click on the arrow next to the “General Resources” and then choose THSS Materials. A pop up window with a downloadable list of materials will appear which includes:
this presentation “Overview of Scoring ELA Interims.ppx”
grade level zip files.
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

11. Preparation: Using THSS materials
Download materials from TIDE:
Grade-level zip files
Reference document (labeled “InterimItems2016”)
Training Guide (labeled “SG”)
Exemplars (labeled “TM”)
Note: Materials also available in the THSS while scoring but only for the particular item being scored
The THSS materials help prepare educators to score student responses from the Smarter Balanced Interim Assessments (IABs, and ICAs).
The materials are located in zip files by grade level. They include a reference guide for the grade level and Training Guide (labeled “SG”) and Exemplars (labeled “TM”) for each hand-scored item.
These materials (Training Guide and Exemplars) are also accessible in the THSS after students have completed the interim assessment, but only for the particular item being scored.
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

12. Download materials from TIDE
In TIDE: located in zip files by grade level
Reference document (labeled “InterimItems2016”)
Training Guide (labeled “SG”)
Exemplars (labeled “TM”)
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

13. Reference Guide—Zip folder
The purpose of the Reference Guide is to provide a summary of the various items with the following: item number, title, claim, target, item stem, and test selection name. This can be used for planning purposes.
9/17/2018

14. Training Guide—Zip folder
There are also Training Guides (labeled “SG”)
9/17/2018

15. Exemplars—Zip folder
And Exemplars (labeled “TM”) for each hand-scored item
9/17/2018

16. Training Guides for ELA
9/17/2018
Training Guides for ELA
Instructions
Stimulus
Item stem
Target specific rubric
Condition codes
Target chart
Each training guide contains item specific stimulus, item stem, target specific rubric, condition codes and a target chart. The condition codes are the only part that is the same for reading, research, and brief writes; the full write codes are a more extensive list.
Instructions: Provide step by step directions for using the materials
Stimulus: The passage or text to which students are asked to respond
Item stem: Task students are asked to complete
Target specific rubric: Use to score student responses
Condition codes: Use during scoring to identify reasons that responses are not scorable
Target chart: Table that shows claim, targets and standards
Note: Due to the animation of this slide, all pages of the document above may not appear when this presentation is printed.
Washington State Office of Superintendent of Public Instruction
9/17/2018
Washington State Office of Superintendent of Public Instruction

17. Steps for Using Training Guides (SG)
Start by reading the preamble (brief writes), stimuli, and accompanying item stem.
Examine the target-specific rubric for each score point.
Read through the supporting documents:
Condition code document―to better understand how to score unusual responses;
Target chart―to support the understanding of all targets in each Claim.
Next, move on to the Exemplars Packet.
Reading the Training Guide is the first step in the process of training yourself (or a group of educators) to score an interim assessment item.
Washington State Office of Superintendent of Public Instruction
9/17/2018

18. Exemplars for ELA Instructions Metadata: Anchor set Check set Key
9/17/2018
Exemplars for ELA
Instructions
Metadata:
Claim
Target
Standards
Writing/Stimulus Type
Anchor set
Check set
Key
The exemplars contain many examples of scored student responses. Reading and understanding the exemplar is the most important part of training yourself (or a group of educators) to score interim assessment items. The Anchor Set of responses are arranged from highest scoring to lowest scoring and are used to help explain how the rubric is applied to student work. The Check Set of responses are randomly ordered and are used to help check your understanding of the rubric.
Exemplars Packet
A document containing an anchor set, a check set, and the key for scoring the check set is located in the Exemplar tab.
The anchor set contains approximately 10 examples of student responses and, when available, scoring comments.
The anchor set is provided to help the scorer understand how the rubric and scored papers align.
Responses are ordered high-to-low for score points 2, 1, and 0.
The check set contains approximately 10 randomly ordered example student responses without scores or scoring comments.
The check set is an opportunity for the scorer to practice accurate scoring.
The score points and scoring comments for the check set can be found in the key.
By viewing the Training Guide and practicing using the Exemplars Packet, educators are ready to begin scoring assigned student responses.
Note: Due to the animation of this slide, all pages of the document above may not appear when this presentation is printed.
Washington State Office of Superintendent of Public Instruction
9/17/2018
Washington State Office of Superintendent of Public Instruction

19. Steps for Using Exemplars (TM)
Read through the anchor set of student responses and scores to see how rubric and scored papers align.
Read through the check set of random student responses.
Practice assigning scores using rubric.
Compare your scores to the key.
Next, you are ready to score your assigned student responses.
Washington State Office of Superintendent of Public Instruction
9/17/2018

20. Step 2: Administration and Scoring
Test administration
Interim teacher hand scoring system (thss)
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

21. Test Administration
Use the Test Administration application to administer interims
For complete directions see the TDS Interface Module
Use the Test Administration interface to open Interim test sessions. Students will use the secure browser to take the test.
Please view the TDS Interface Module located in the Modules folder of the Test Administration Resources section of the WCAP Portal for complete information.
After student tests have been completed and submitted you can move on to scoring their responses.
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

22. Scoring Student Responses
Log in to the Interim Teacher Hand Scoring System
Filter to organize responses
Dropdown menus: Test, session, grades, subjects, or scorers
Green heading: Student name, item, session, status, or assigned to

23. Exemplar and Training Guide
Student Response Page
Rubric
Enter the score
Task or item
Select “Score” to view the individual student response page. The Exemplar and Training Guides are located here (the same ones found in TIDE), a rubric, a drop down to enter the score, a condition code (information about the condition code can be found in the Training Guide), the item stem, and the student’s response. You enter the score in the middle of the screen, and then submit the score at the bottom of the screen.
Student’s Response
Submit score

24. Response List: Completing Item Responses
When you have finished scoring, you must check the box next to the student's name on the Score Response page and then click the button at the bottom of the page called Mark Selected as Complete.
Once an item is marked complete it is submitted and can no longer be edited or viewed. It will no longer appear on this list.
Once all of a student’s items for a specific test have been scored and marked complete the student’s score for the entire test will be available in the Online Reporting System and in AIR Ways. If you get into ORS or AIR Ways and don’t see data for a student, go back to THSS and make sure none of there responses are sitting on this list.

25. Appropriate use of materials
This presentation may be shared with any audience
Contains no secure information
Also available in the WCAP Portal
Training Guides and Exemplars may be used for these audiences:
Individuals – teachers, parents, and/or students
Groups – during a PLC, department, or school-wide meeting
Contains non-secure, but non-public information: Do not distribute
Use this presentation with interested stakeholders.
It is also available in the General Information folder of the Test Administration Resources portion of the WCAP Portal.
Training Guides and Exemplars may be used with individuals and/or with groups, such as PLCs, departments, or at a school-wide meetings. These materials are considered non-secure but also non-public. These materials are intended for training and scoring purposes only and should not be distributed for other purposes.
Teachers could use the materials with students to help them understand expectations; however, students should not receive copies or distribute any of these materials.
Important to know:
Designated as “non-secure* and non-public*”
*Non-secure means access to the items, answers, and student responses – for educators and students (also not intended for accountability purposes)
*Non-public means they are not be posted publicly (such as a district or teacher website)
Optional to allow Local districts to establish timeframe, policies around usage, and scoring practices
Important that districts/schools/teachers use discretion to check progress of specific concepts at strategic points during the year
Washington State Office of Superintendent of Public Instruction
9/17/2018

26. Step 4: Reflection Online reporting system (ORS) Air ways reporting
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

27. Reflection
“Formative Assessment is not a test but a process—a planned process involving a number of different activities. One of those activities is the use of assessments, both formal and informal, to elicit evidence regarding students’ status: the degree to which a particular student has mastered a particular skill or body of knowledge.”
–Jim Popham, Transformative Assessment
Use data from ORS and AIR Ways to reflect on student learning in order to improve teaching and learning
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

28. AIR Ways Reporting:
Purpose: Use data to determine strategies to improve teaching and learning for classes, groups, and/or individual students
Performance data:
District
School
Class
Student
Information presented:
Item, answer key/rubrics
Actual student responses (including hand scored items)
Item difficulty and frequency distribution
Content alignment (target)
Formerly known as Interim Reports Viewer (IRV) that was beta tested last year, AIR Ways reporting is used for viewing interim results.
Keep in mind:
Students need to take interims first
If any hand scoring is required, that scoring must be completed prior to reports populating in AIR Ways Reporting
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

29. AIR Ways Reporting: Features
Expand
Data can be viewed by clicking the plus to expand the columns to show information for the items on which students performed the best, the worst, and all items.
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

30. AIR Ways Reporting: Features
Item number
Points possible
Average
Student scores:
Click the underlined numbers to show the interim items
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

31. AIR Ways Reporting: Features
Displays 5 items for which student performance was the highest
Displays 5 items for which student performance was the lowest
Identifies 5 items within an interim assessment for which student performance was the lowest
Identifies 5 items within an interim assessment for which student performance was the highest
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

32. AIR Ways Reporting: Details displayed
Details for an item:
Frequency distribution of scores
Difficulty estimate
Content alignment: Target language
Item with answer choices and student’s response
Correct answer/rubric
Details for an item:
Frequency distribution of scores (how many 0, how many 1)
Difficulty estimate (based on summative field test/operational data): e.g., Easy, Moderate, Difficult
Content alignment: target language
Item with answer choices
Item with student response (both MC and constructed)

33. AIR Ways Reporting: Details displayed
Details for a student:
Comparison to “Everyone”
Item with student’s response
Student performance descriptor
Student’s information
Details for a student:
Comparison to “everyone” in the class
Item with student response
Student performance descriptor: e.g. Below, At/Near, Above
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

34. Comparing ORS to AIR Ways:
Both systems…
Show overall performance for classroom, school, district on interim assessments, including average score and percent in each proficiency category
Provide the point value for each interim item
Indicate the skill being assessed–by assessment target
Show all student opportunities
Have export and print capabilities
For exporting and printing note that CSV files = Excel spreadsheet
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

35. Comparing ORS to AIR Ways
Specific to Ors
Specific to air ways
Can view reports for ALL state assessments, all subjects
Provides combined classroom, school, district performance for each interim item and/or targets
Has trend reports
Has family and student friendly reports
Interim item data only
Shows each original item and also individual student responses
Includes scoring rubrics and correct responses
Sorts the items by 5 best/5 worst for a group of students
Shows:
-each item as presented to students (blank) and as submitted by students (with their answer included)
-scoring rubric
-breakdown of ICA items by target or claim
Sorts the items on which students/rosters did best and worst
Allows the educator to organize tests by “test reason” for multiple opportunites (COMING SOON)
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
9/17/2018

36. For Further Information
Shelley O’Dell– ELA Assessment Specialist
Terese Otto – ELA Administrative Support
User Guides:
Test Administrators - User Guides and Manuals
THSS User Guide
ORS User Guide
AIR Ways User Guide
Washington State Office of Superintendent of Public Instruction
9/17/2018

37. Revision Log Updates to the 2016-2017 are noted below. Section Slide
Description of Change
Revision Date
AVA 6
Added information about AVA
2/3/2017
THSS Materials
11-19
Expanded information about THSS materials that are in TIDE
AIR Ways
28-35
Added information about AIR Ways
Throughout the year, if updates are needed to this module, the changes will be detailed in the revision log on the PowerPoint. Any changes will be communicated in the Washington Assessment Weekly newsletter as well