1. Using empirical feedback to develop a learning progression in science Karen Draney University of California, Berkeley

2. 2 The components of the BEAR Assessment System

3. Measurement model provides empirical feedback at every step Theoretical structure of learning progression Quality of items Match between item scoring process and learning progression Degree to which assessment is an accurate reflection of student proficiency

4. Carbon Cycle as example -- two of the original constructs

5. Wright map (Note: “X’ = 1 respondent.)

6. Original scoring with 5 levels 6 Quite difficult to reach second level Relatively easy to reach second level

7. Latest structure: Learning progression for Carbon

8. Item thresholds

9. CHARACTERISTICS OF RESPONSES CRD: “A carbon atom from Grandma Johnson's remains sink into the ground and mixes with the soil. Then when the soil is mixed and churned, it rises to the top of the ground. When the coyote kills something upon that dirt, he may consume it and have some of the soil come with it, which produces him with Grandma Johnson's carbon atom.” MCK: “The carbon in grandma’s body is decomposed into the ground. The plants then use the fertile soil to use her carbon atoms. As the soil passes it to the plant, the plant is eventually eaten by the coyote. The carbon atom then travels to its leg.” DES: “Grandma Johnson's remains decay and decomposers use respiration and turn it to carbon dioxide. The plants absorb the carbon dioxide. Rodents eat the plants and then the coyote eats the rodent.” Written Response Items Grandma Johnson: Describe the path of a carbon atom from Grandma Johnson’s remains, to inside the leg muscle of a coyote. NOTE: The coyote does not dig up and consume any part of Grandma Johnson’s remains. Tracing Matter: Material Kinds, Properties of Matter 5 Correctly identifies reactants and products of individual chemical processes: Substances/molecules and the elements/atoms of which they are composed. 4 Correctly identifies some reactants and products of single chemical process, and solids, liquids, and gases involved. 3 Correctly identifies some reactants and products of single chemical process, and solids, liquids, but not gases involved. 2 Attention to hidden mechanism, but cannot identify any material kinds 1 Identify changes by common sense of natural phenomena, but not as changes in materials and cannot identify material kinds.

10. Other item types

11. Item fit - how well are items working More random noise than expected in this item

12. xxx xxxx xx xxxxx xxxxxx xxxxxxxx xxx xxxxxx xxxxx xxxxxxxxxxx xxxxxxxxx xxxxxx xxxxxxx xxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxx xxxxx xxxxxxxxxxxxxx xxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxx xxxxxxxxxx xxxx xx x 43214321 43214321 43214321 Item difficulty estimates Student performance estimates 43214321 Q1 Q2 Q3 Q4 x = one student 2.0 1.0 0 -2.0 one scale Match between scoring and learning progression

13. Person fit - how well does the assessment work for individuals