Kentucky’s Experience with Three-Dimensional Science Assessment National Conference on Student Assessment Philadelphia Marriott June 21, 2016

Session Presenters Stephen Pruitt—Commissioner of the Kentucky Department of Education (KDE) Karen Kidwell—Director of the Division of Program Standards (KDE) Roger Ervin—System Administrator for the Office of Assessment and Accountability (KDE) Nicolle Romero—Science Assessment Development Manager, WestEd Joanne Jensen—Director of Assessment Client Relations, WestEd

Kentucky’s Guiding Principles Equity Achievement Integrity

Science Assessment: A Team Process The Commissioner Office of Teaching and Learning Office of Assessment and Accountability

A SYSTEM of Science Assessments Built on teaching and learning in classroom Each component adds and elicits new evidence of students’ learning Defensible evidence of student attainment of the standard, as the standard was intended, is generated Supports different purposes/users/uses Classroom Embedded Assessments Statewide Summative Assessment Through-Course Tasks PE

Leverage THINKING PARTNERS for Quality Started with BOTA report to conceptualize vision of system Supported by Commissioner to seek expert advice Established an RFP for a THINKING PARTNER separate from/ahead of seeking a vendor Simultaneously supported in-field teachers/administrators with deep, ongoing professional learning around the KAS for Science (3 years)—and the vision for the system – taking their feedback into account Balanced involvement of ‘assessment’ and ‘content’ specialists in bringing the vision to reality

Lessons Learned (so far!) The WHOLE (of PEs) is greater than the sum of their parts Seek clarity on the role of the PEs and all other supporting resources available for the standards Storylines bring coherence and purpose to item clusters/tasks 3-Dimensionality is challenging but not impossible Balance desire to innovate on ‘items’ with the reality/need to try things out in order to make progress Plan for investment in professional learning to support classroom change Must rethink the ‘metrics’ of scoring/reporting with clusters

Role of Assessment and Accountability Science Assessments Balancing Act Supporting the goals of the Commissioner, Kentucky Board of Education and the Department related to science education Connecting program staff and vendor services Building for appropriate administration of state-required tests Goals How to ensure Kentucky students have a proper learning trajectory for science and become science literate? How can assessment support quality science instruction and curriculum? Challenges What is doable within limits of technologies, vendor capacity, time and money? Science Assessment Plan

Science Assessment - Goals Measure the richness of the standards (i.e., content spectrum and three dimensions) Provide for student accommodations and consistent test administration Support student attainment of science standards at every grade throughout the school year (i.e., classroom items and tasks) Develop a robust and sustainable bank of items/clusters Yield valid and reliable student/school performance results Report detailed results at the school level Develop potentially online interactive assessments (i.e., simulations and tasks)

Science Assessment - Challenges Every Student Succeeds Act (ESSA) U.S. Department of Education interpretation of the law State interpretation of the law Kentucky State Laws All state summative assessments (i.e. reading/ELA, on-demand writing, mathematics, science and social studies) administered in a five-day window in the last 14 days of the school Instructional - calendar Limitations on testing time (2 – 2½ hours) Timelines for reporting student and school performance results Limited money $$$

Science Assessment – Challenges Paper/pencil assessments Does not easily allow support for technology-based accommodation solutions Costs for printing and shipping are increasing; processing can be slowed and test results delayed Technology-based assessments Technology is constantly changing Considerations need to be given to schools with technology limitations versus schools with one-to-one technology for students Vendor concerns include: reliable delivery of an online assessment, limitations of processing platforms and limited experience with effectively supporting simulations and tasks

Science Assessment Plan For the first time, science test development is grounded in science phenomenon On-going approach to item and item cluster development to allow for continuous improvement of the test and development of teacher and administrator science literacy Test administration – measure richness of standards across years Field test new items/item clusters each year to refresh a robust and sustainable bank Paper and pencil assessments initially; however, building items with an eye toward future technology and use of online interactive platforms that may include simulations and tasks

Summative Assessment Plan Elementary and middle school Alternate assessment (grades 4 and 7) Full operational test in 2016-17 Standard setting in summer 2017 Report student performance levels to students, parents and schools/districts in fall 2017 Not included in accountability until new system in 2017-18 Regular assessment (grades 4 and 7) Field test administered in 2016-17 Goal to build item bank No standard setting or reporting of student performance levels in 2016-17

Science Assessment Plan High school assessment TBD but not end-of-course (EOC) (Biology EOC measures only part of science standards) Option 1 – Summative assessment Option 2 – Assessment modules that cover student’s high school experience Option 3 – Hybrid of Options 1 and 2

Accountability Historically, Kentucky has included science performance in state accountability New accountability system under development

State Summative Assessment (SSA) Prototype Demonstrates a starting point for the vision and expectations for the summative assessment of the Kentucky Science Assessment System A collaborative effort with KDE staff, science content experts, and assessment designers and developers Intended uses: To serve as an initial model for measuring the three-dimensional science learning as called for with the NGSS (NGSS Lead States, 2013) and in A Framework for K–12 Science Education (hereafter “the K–12 Framework”) (NRC, 2012). To support Kentucky in guiding its test vendors with the design and development of the three dimensional-aligned state summative assessment portion of the Kentucky Science Assessment System.

State Summative Assessment (SSA) Prototype The SSA prototype is aligned to a PE bundle and presented within the context of a phenomenon. The purposes of PE bundles are: (1) to ensure that PEs are not assessed in isolation from one another (as recommended in the BOTA report; NRC, 2014); (2) to allow for an assessment to assess as broad a range of PEs as possible without compromising validity; and (3) to support the identification of appropriate phenomena that would provide a basis for assessing the given PEs.

State Summative Assessment (SSA) Prototype

State Summative Assessment (SSA) Prototype

State Summative Assessment (SSA) Prototype

Science Standards Alignment Considerations The basis of a PE bundle is the individual PEs and the supporting information found within the PE foundation boxes. Item clusters must be determined as meeting the intent of the PEs (i.e., Kentucky Academic Standards (KAS) for Science) to which they are aligned. The intent of the PEs is informed by any item specifications developed by KDE and by analyses based on the information specified in the PE foundation boxes and additional documentation (e.g., K–12 Framework, NGSS Progression Appendices).

Science KAS Alignment Considerations

Science KAS Alignment Considerations Context judgments PE bundle (appropriate? yes/no judgment) Phenomenon (appropriate? yes/no judgment) Stimulus (appropriate? yes/no judgment) Individual item judgments Item to SEP (0–3 alignment judgment) Item to DCI (0–3 alignment judgment) Item to CCC (0–3 alignment judgment) Item to PE (0–3 alignment judgment) Overall item cluster judgments Item cluster to PE (appropriate? yes/no) PE bundle to item cluster (appropriate? yes/no) Item cluster (acceptable for use? yes/no)

Kentucky Spring 2016 Pilot 4 ‘versions’ of a cluster based on same storyline 13 student responses required (MC, short answer, matching, extended response) 1300 7th grade students; 26 teachers/classrooms 40 individual cognitive lab interviews (10 each version) 1 class period (from 40 – 75 minutes) No accommodations were made for the pilot

Kentucky Spring 2016 Pilot Findings ~25% of students successfully completed claim, evidence, reasoning items – when presented as MC; fewer succeeded when they were written responses Interpreting and ‘completing’ a model was very challenging; ~5% of students were successful with all elements across all versions (Though in cog labs, several students could explain the model correctly while still not completing it correctly) “Mark all that apply” presented difficulty; most students got at least ONE of the correct choices, yet few marked both correctly

Kentucky Spring 2016 Pilot Findings, cont. Use of technology to support reading, add additional graphics, enhance the storyline through video was not generally viewed as an enhancement to students The storyline did engage students – helped them see the point of the questions The reading load of the storyline was challenging (time, length, complexity) Teachers viewed the structure and ‘content’ of the cluster as significantly more challenging than other science assessments

Contact Information Stephen L. Pruitt Karen Kidwell Commissioner of Education Director Division of Program Standards Kentucky Department of Education Kentucky Department of Education Stephen.Pruitt@education.ky.gov Karen.Kidwell@education.ky.gov Roger W. Ervin Nicolle Romero Systems Administrator Science Assessment Development Manager Kentucky Department of Education and Technology Specialist, WestEd Roger.Ervin@education.ky.gov nromero@wested.org Joanne Jensen Director of Assessment Client Relations WestEd jjensen@wested.org