1. Kentucky’s Experience with Three-Dimensional Science Assessment
National Conference on Student Assessment
Philadelphia Marriott
June 21, 2016

2. Session Presenters
Stephen Pruitt—Commissioner of the Kentucky Department of Education (KDE)
Karen Kidwell—Director of the Division of Program Standards (KDE)
Roger Ervin—System Administrator for the Office of Assessment and Accountability (KDE)
Nicolle Romero—Science Assessment Development Manager, WestEd
Joanne Jensen—Director of Assessment Client Relations, WestEd

3. Kentucky’s Guiding Principles
Equity
Achievement
Integrity

4. Science Assessment: A Team Process
The Commissioner
Office of Teaching and Learning
Office of Assessment and Accountability

5. A SYSTEM of Science Assessments
Built on teaching and learning in classroom
Each component adds and elicits new evidence of students’ learning
Defensible evidence of student attainment of the standard, as the standard was intended, is generated
Supports different purposes/users/uses
Classroom Embedded Assessments
Statewide Summative Assessment
Through-Course Tasks PE

6. Leverage THINKING PARTNERS for Quality
Started with BOTA report to conceptualize vision of system
Supported by Commissioner to seek expert advice
Established an RFP for a THINKING PARTNER separate from/ahead of seeking a vendor
Simultaneously supported in-field teachers/administrators with deep, ongoing professional learning around the KAS for Science (3 years)—and the vision for the system – taking their feedback into account
Balanced involvement of ‘assessment’ and ‘content’ specialists in bringing the vision to reality

7. Lessons Learned (so far!)
The WHOLE (of PEs) is greater than the sum of their parts
Seek clarity on the role of the PEs and all other supporting resources available for the standards
Storylines bring coherence and purpose to item clusters/tasks
3-Dimensionality is challenging but not impossible
Balance desire to innovate on ‘items’ with the reality/need to try things out in order to make progress
Plan for investment in professional learning to support classroom change
Must rethink the ‘metrics’ of scoring/reporting with clusters

8. Role of Assessment and Accountability Science Assessments
Balancing Act
Supporting the goals of the Commissioner, Kentucky Board of Education and the Department related to science education
Connecting program staff and vendor services
Building for appropriate administration of state-required tests
Goals
How to ensure Kentucky students have a proper learning trajectory for science and become science literate?
How can assessment support quality science instruction and curriculum?
Challenges
What is doable within limits of technologies, vendor capacity, time and money?
Science Assessment Plan

9. Science Assessment - Goals
Measure the richness of the standards (i.e., content spectrum and three dimensions)
Provide for student accommodations and consistent test administration
Support student attainment of science standards at every grade throughout the school year (i.e., classroom items and tasks)
Develop a robust and sustainable bank of items/clusters
Yield valid and reliable student/school performance results
Report detailed results at the school level
Develop potentially online interactive assessments (i.e., simulations and tasks)

10. Science Assessment - Challenges
Every Student Succeeds Act (ESSA)
U.S. Department of Education interpretation of the law
State interpretation of the law
Kentucky State Laws
All state summative assessments (i.e. reading/ELA, on-demand writing, mathematics, science and social studies) administered in a five-day window in the last 14 days of the school Instructional - calendar
Limitations on testing time (2 – 2½ hours)
Timelines for reporting student and school performance results
Limited money $$$

11. Science Assessment – Challenges
Paper/pencil assessments
Does not easily allow support for technology-based accommodation solutions
Costs for printing and shipping are increasing; processing can be slowed and test results delayed
Technology-based assessments
Technology is constantly changing
Considerations need to be given to schools with technology limitations versus schools with one-to-one technology for students
Vendor concerns include: reliable delivery of an online assessment, limitations of processing platforms and limited experience with effectively supporting simulations and tasks

12. Science Assessment Plan
For the first time, science test development is grounded in science phenomenon
On-going approach to item and item cluster development to allow for continuous improvement of the test and development of teacher and administrator science literacy
Test administration – measure richness of standards across years
Field test new items/item clusters each year to refresh a robust and sustainable bank
Paper and pencil assessments initially; however, building items with an eye toward future technology and use of online interactive platforms that may include simulations and tasks

13. Summative Assessment Plan
Elementary and middle school
Alternate assessment (grades 4 and 7)
Full operational test in
Standard setting in summer 2017
Report student performance levels to students, parents and schools/districts in fall 2017
Not included in accountability until new system in
Regular assessment (grades 4 and 7)
Field test administered in
Goal to build item bank
No standard setting or reporting of student performance levels in

14. Science Assessment Plan
High school assessment
TBD but not end-of-course (EOC) (Biology EOC measures only part of science standards)
Option 1 – Summative assessment
Option 2 – Assessment modules that cover student’s high school experience
Option 3 – Hybrid of Options 1 and 2

15. Accountability
Historically, Kentucky has included science performance in state accountability
New accountability system under development

16. State Summative Assessment (SSA) Prototype
Demonstrates a starting point for the vision and expectations for the summative assessment of the Kentucky Science Assessment System
A collaborative effort with KDE staff, science content experts, and assessment designers and developers
Intended uses:
To serve as an initial model for measuring the three-dimensional science learning as called for with the NGSS (NGSS Lead States, 2013) and in A Framework for K–12 Science Education (hereafter “the K–12 Framework”) (NRC, 2012).
To support Kentucky in guiding its test vendors with the design and development of the three dimensional-aligned state summative assessment portion of the Kentucky Science Assessment System.

17. State Summative Assessment (SSA) Prototype
The SSA prototype is aligned to a PE bundle and presented within the context of a phenomenon.
The purposes of PE bundles are:
(1) to ensure that PEs are not assessed in isolation from one another (as recommended in the BOTA report; NRC, );
(2) to allow for an assessment to assess as broad a range of PEs as possible without compromising validity; and
(3) to support the identification of appropriate phenomena that would provide a basis for assessing the given PEs.

18. State Summative Assessment (SSA) Prototype

19. State Summative Assessment (SSA) Prototype

20. State Summative Assessment (SSA) Prototype

21. Science Standards Alignment Considerations
The basis of a PE bundle is the individual PEs and the supporting information found within the PE foundation boxes.
Item clusters must be determined as meeting the intent of the PEs (i.e., Kentucky Academic Standards (KAS) for Science) to which they are aligned.
The intent of the PEs is informed by any item specifications developed by KDE and by analyses based on the information specified in the PE foundation boxes and additional documentation (e.g., K–12 Framework, NGSS Progression Appendices).

22. Science KAS Alignment Considerations

23. Science KAS Alignment Considerations
Context judgments
PE bundle (appropriate? yes/no judgment)
Phenomenon (appropriate? yes/no judgment)
Stimulus (appropriate? yes/no judgment)
Individual item judgments
Item to SEP (0–3 alignment judgment)
Item to DCI (0–3 alignment judgment)
Item to CCC (0–3 alignment judgment)
Item to PE (0–3 alignment judgment)
Overall item cluster judgments
Item cluster to PE (appropriate? yes/no)
PE bundle to item cluster (appropriate? yes/no)
Item cluster (acceptable for use? yes/no)

24. Kentucky Spring 2016 Pilot
4 ‘versions’ of a cluster based on same storyline
13 student responses required (MC, short answer, matching, extended response)
1300 7th grade students; 26 teachers/classrooms
40 individual cognitive lab interviews (10 each version)
1 class period (from 40 – 75 minutes)
No accommodations were made for the pilot

25. Kentucky Spring 2016 Pilot Findings
~25% of students successfully completed claim, evidence, reasoning items – when presented as MC; fewer succeeded when they were written responses
Interpreting and ‘completing’ a model was very challenging; ~5% of students were successful with all elements across all versions (Though in cog labs, several students could explain the model correctly while still not completing it correctly)
“Mark all that apply” presented difficulty; most students got at least ONE of the correct choices, yet few marked both correctly

26. Kentucky Spring 2016 Pilot Findings, cont.
Use of technology to support reading, add additional graphics, enhance the storyline through video was not generally viewed as an enhancement to students
The storyline did engage students – helped them see the point of the questions
The reading load of the storyline was challenging (time, length, complexity)
Teachers viewed the structure and ‘content’ of the cluster as significantly more challenging than other science assessments

27. Contact Information
Stephen L. Pruitt Karen Kidwell
Commissioner of Education Director Division of Program Standards
Kentucky Department of Education Kentucky Department of Education
Roger W. Ervin Nicolle Romero
Systems Administrator Science Assessment Development Manager
Kentucky Department of Education and Technology Specialist, WestEd
Joanne Jensen
Director of Assessment Client Relations
WestEd