Item pool optimization for adaptive testing Marty McCall Director of Psychometrics, Smarter Balanced National Conference on Student Assessment June, 2016 - Philadelphia, PA

Construct modeling and test design Content specifications derived from CCSS Assessment standards modeled on instructional standards Grade-to-grade progressions modeled Test structure follows content specs Item specifications address specific content spec elements Test blueprints derived from content specifications Depth and balance Time/cost constraints

Test Blueprint What do you want every student to get? Content – categories and proportions Cognitive characteristics Item types How many items in each test event? What are you going to report? For individuals? For groups? Overall scores Sub-scores Achievement category

Adaptive test: Item pool + test blueprint + algorithm Constrained CAT Test blueprint is a design for the student test event. It specifies the content for every student taking the test Item pool distributed to supply tests at every score level The algorithm uses the item pool to find the most informative test for each student within blueprint constraints

Pool design Reckase (2007) – P-optimal pool evaluation. “It’s unrealistic to expect that every value of theta will have a maximally informative item.” p-optimal means assuring an item within a specified proportion of full optimality for each item requested by the algorithm. Theta-scale is represented as adjacent “bins” Width of bins determined by level of optimality for desired SEM

Smarter Adaptive Test Blueprint Hierarchical relationships Claims ELA-1. Reading, 2. Writing, 3. List/Spk, 4. Research Math-1. Concepts/Procedures, 2. Problem Solving, 3. Communicating Reasoning, 4. Modeling/dData Analysis Target Clusters Targets Global rules DOK item type passage rules(ELA) Claims consistent across grades, targets consistent across grades in ELA, vary in math

Mathematics, Grade 5 Blueprint Target Specifications

Mathematics, Grade 5 Blueprint Target Specifications

Examples of non-hierarchical rules In grades 6-8, up to one CAT item per student may require hand-scoring (from either Claim 3 or Claim 4 Claim 2 (Problem Solving) and Claim 4 (Modeling and Data Analysis) have been combined .... There are still four claims, but only three claim scores will be reported with the overall math score. DOK: The CAT algorithm will be configured to ensure the following: For Claim 1, each student will receive at least 7 CAT items at DOK 2 or higher. For combined Claims 2 and 4, each student will receive at least 2 CAT items at DOK 3 or higher. For Claim 3, each student will receive at least 2 CAT items at DOK 3 or higher.

Item pool specification sheet; aka The Beast

Mathematics, grade 5 Claim 1 Beast Detail Mathematics, grade 5 Claim 1 Total N broken out by the number of bins

Bins Bin boundaries are artificial (Van der Linden method continuous, but similar) Item writers can’t predict at the fine levels required by bins They can respond to student levels Made estimates of easy, medium and hard In general, underestimated item difficulty, which has caused us to examine scale score characteristics through item mapping

Expanded pools Pools are evaluated by grade level For students at high or low extremes, the pool expands to adjacent grade levels to provide better measurement After about 2/3 of test has been administered For students who are far above or below standard Pool includes items from adjacent grades that content experts have judged to be measuring the grade level target Items with the most information are chosen from expanded pool

Pool evaluation and maintenance Uses the bin method to estimate needs Compares needs to operational pool Gives recommendations for item writing, item retirement, etc Data structure connected to the Item database for constant updating Clear and practical method

Thank you for your attention Questions? marty.mccall@smarterbalanced.org