1. Smarter Balanced Achievement Level Setting
Gil Mendoza, Ed.D.
Deputy Superintendent

2. Smarter Balanced – Preparing for New Test Scores
9/22/2018

3. 2015 results will set a new baseline of student performance in Washington
Some people are predicting that the spring 2015 test results will see Washington students falling off a “performance cliff”, with scores dropping.
But the new tests are
not comparable so we
should not compare
scores from one to the
other.
We have a new baseline!

4. 2015 results, cont’
Think of the standards and the assessment as a new target with new results…. I envision two mountains:
People who successfully climb Mt Rainer (at 14,000 ft), will find Mt McKinley (at 20,000 ft) more challenging.
Some will be able to meet the challenge, some will be close and some who previously were able to summit Rainier will not be able to summit McKinley at first.

5. Preparing for New Test Scores
Smarter Balanced assessments measure the full range of the Common Core State Standards. They are designed to let teachers and parents know whether students are on track to be college- and career-ready by the time they graduate.
Because the new standards set higher expectations for students--and the new tests are designed to assess student performance against these higher expectations--our definition of grade level performance is higher than it used to be.
As a result, it’s likely that fewer students will meet grade level standards, especially for the first few years. Results should improve as students have additional years of instruction aligned to the new standards and become better equipped to meet the challenges they present.
This does not mean that our students are “doing worse” than they did last year. Rather, the scores represent a “new baseline” that provides a more accurate indicator for educators, students, and parents as they work to meet the rigorous demands of college and career readiness.
Since the new assessments are set to higher standards, the definition of grade level performance is also higher than it was with old tests. This is the path to raising the bar for students and achieving college and career readiness.
Yet for the first few years, the percentage of students meeting grade level standards is likely to be much lower. Without the proper context, this could easily be misinterpreted to mean that students and schools are not performing as well as they did previously.
It is critical to communicate with teachers, parents, and students about what the new test scores mean in advance of their release.
These key messages have been developed with input from parents and teachers.
It’s important to incorporate them into school and teacher communications with parents:
At the beginning of the school year;
When the assessments are administered;
When results are released.

6. Smarter Balanced Achievement Level Setting Activities
Online Panel
Over 10,000 registered
5,840 logged in
2,660 submitted
In-Person Panel
504 panelists confirmed
482 completed the tasks
Vertical Articulation Committee
64 panelists confirmed
CTB staff, working with MI, Hager Sharp, and Smarter Balanced, continued to recruit for the Online Panel through October 13. Just over 10,000 people signed up to review one test.
CTB and MI staff recruited all 504 in-person panelists and 10 alternates. We then recruited 72 more alternates and replaced panelists as we received last-minute cancellations.
We also confirmed VAC 64 panelists, all but one of whom were able to serve. We replaced the one VAC panelist on-site.
Our biggest challenge was the Ebola case that hit the news on September 30. We implemented our contingency communication plan and worked with our contacts to replace panelists who dropped out. We continued to receive cancellations even after we arrived on site in Dallas. 6

7. Washington’s Participation
For the Online Panel Washington had 1,293 total participants:
916 – Teachers
238 – Other educators (admin and higher ed)
  86 – Parents
  53 – Other (business/community members)
For In Person Panel= 30, mostly teachers but a few admin and higher ed
Smarter Balanced is particularly interested in participation by K–12 educators and higher education faculty. There are considerable benefits to participating in the Online Panel.
Additional information:
Cost: Participation is free.
Privacy: Personally identifiable information of individual participants will not be released.
Accessibility: Accessible through virtually any computing device that connects to the Internet, including tablets (Smart Phones not recommended due to screen size).
Reminder: Panelists will receive reminders before their two-day window opens.
Time Commitment: Up to 3 hours over a two-day window. Ability to save work and log out, and then log back in to complete session within the two-day window.

8. Achievement Level Setting Participation (across states)
Online Panel
In-Person Panel
Cross-Grade Review Committee
Teachers
65%
64%
66%
Non-Teacher Educators & Administrators
18%
19%
Higher Education 9%
12%
11%
General Public 7% 6% 5%
Total
2,660
482 64
EDUCATOR INVOLVEMENT:
The levels were set by teachers and practitioners who understand student needs.
The recommendations were developed with input from thousands of educators and community members, reflecting a diverse cross-section of views on education.
Throughout this process, Smarter Balanced involved educators and other stakeholders and to account for many diverse views in the fields of education and assessment.

9. Smarter Balanced Scale Scores
Scale scores on Smarter Balanced assessments
Use a four digit number
Range roughly from 2000-to-3000
Avoids confusion with other, commonly-used scales such as: percentile ranks, percent correct, ACT, SAT, Lexiles, etc.
Provides adequate “space” on the scale to see growth in whole-number units

10. Approved Initial Cut Scores - ELA

11. Approved Initial Cut Scores - Math

12. Consortium-wide Impact of Cut Scores
41% met standard in 11th

13. Consortium-wide Impact of Cut Scores
33% met standard in 11th

14. Percent of Field Test Sample That Met Standard (Scored Level 3 or Level 4)
Grade
ELA
Math 3
38%
39% 4
41%
37% 5
44%
33% 6 7 8
32% 11

15. Smarter Balanced + Higher Ed
Washington has had unprecedented movement to use mandatory high school tests in college placement
Agreement to provide access to credit bearing courses to 11th graders who score a 3 or 4 on the Smarter Balanced ELA and math tests
Washington’s Community Colleges
Washington’s Baccalaureates

16. What about Graduation?
Legislature has given the State Board the option to set a different cut score for graduation purposes
The ELA exit exam cut score will be determined from 10th graders’ performance on the Smarter Balanced HS ELA test (for classes of , at least).
Math EOC exit exam cut scores will be determined from this year’s performance on new EOC tests (for classes of ). Exit exam cut score on Smarter Balanced math may also be set next summer – SBE discussion at November meeting.
It’s not too different from deciding how many points students have to earn to get an A, a B, a C, and so on. It’s basically a way of taking a large continuum of scores and dividing it into discrete categories. Of course, as with grades, where there may not be much difference between the highest C and the lowest B, differences among students very close to but on different sides of a cut score may be rather small.

17. College and Career Ready Cut Scores vs High School Graduation Cut Scores
From Draft Position Statement:
As directed by the Legislature, with the assistance of the Superintendent of Public Instruction, the Board hereby intends to adopt a performance level requirement for high school graduation that is statistically equivalent to the current high school minimum proficiency standard, and intends to periodically review this standard over time.

18. Equal Impact
Equal impact cut scores would yield the same “passing” rates on the new tests as the former tests
Reading, Writing or a combination?
2014 or average of past three years?
10th grade pass rates or 11th grade after retakes?
Higher cut scores over time?

19. Where would equal impact fall on the Smarter Balanced scale?
72% of students scored at Level 2 or above on the ELA field test, so to have ~84% meeting in ELA the cut score would be in the Level 1 range.
Likewise, only 60% of students scored at Level 2 or above on the Math field test, so to have ~72% meeting in Math the cut score would be in the Level 1 range.

20. Achievement Levels Level 2 instead of Level 3
New ALD for meeting High School graduation standards

21. Impact of Using Level 2 as Cut Score

22. Impact of Using Level 2 as Cut Score

23. Policy Achievement Level Descriptors
The Level 4 student demonstrates thorough understanding of and ability to apply the English language arts and literacy (mathematics) knowledge and skills needed for success in college and careers, as specified in the Common Core State Standards.
The Level 3 student demonstrates adequate understanding of and ability to apply the English language arts and literacy (mathematics) knowledge and skills needed for success in college and careers, as specified in the Common Core State Standards.
The Level 2 student demonstrates partial understanding of and ability to apply the English language arts and literacy (mathematics) knowledge and skills needed for success in college and careers, as specified in the Common Core State Standards.
The Level 1 student demonstrates minimal understanding of and ability to apply the English language arts and literacy (mathematics) knowledge and skills needed for success in college and careers, as specified in the Common Core State Standards.

24. Ensuring College and Career Readiness Graduating Same # of Students
What about having all students who graduate be college and career ready?
Ensuring College and Career Readiness
Graduating Same # of Students

25. Higher cut scores over time?K
2027
CCR 1
2026 2
2025 3
2024 4
2023
new 5
2022 6
2021 7
2020 8
2019 9
2018 10
2017 11
2016
HSPE 12
2015 NA