1. Analyzing students’ learning performances in terms of practices for developing accounts Hui Jin, Jiwon Kim and Charles W. Anderson

2. Research Goal To develop a teaching-oriented learning progression framework for carbon-transforming processes in socio- ecological systems.To develop a teaching-oriented learning progression framework for carbon-transforming processes in socio- ecological systems. COMMON FRAMEWORK: Learning Progression Framework AssessmentCurriculumInstruction

3. General approach PHASE 1: Use assessment data to develop/revise the learning progression framework PHASE 2: Use the learning progression framework to guide curriculum development; Implement the curriculum and collect interview and written data before and after the teaching intervention PHASE 3: Analyze assessment data and use results of analysis to revise learning progression framework

4. Data sources (2009-10) 2009-10 Data SourceTestsInterviews Elementary Schools16714912 Middle Schools28833816 High Schools26219516 College72871300

5. ASSESSMENT QUESTION 1: What is the scientific reasoning about carbon- transforming processes?

6. Three scientific elementsTwo characteristic ways of scientific reasoning Scientific explanations of carbon- transforming processes MATTER ENERGY SCALE Using principles of matter and energy to constrain accounts Reasoning across scales

7. Using principles of matter and energy to constrain accounts Using Electric appliances Driving Vehicles Burning fossil fuels Body Movement; Dead Organism Body Decay Plant Growth Animal Growth Organic Carbon Oxidation (Combustion) Organic Carbon Oxidation (Cellular Respiration) Organic Carbon Generation (Photosynthesis) Organic Carbon Transformation (Biosynthesis, digestion) CO 2 Light Energy Organic Carbon Chemical Energy Organic Carbon Chemical Energy Organic Carbon Chemical Energy Organic Carbon Heat CO 2 Heat PRINCIPLES Matter Conservation, Energy Conservation, Energy Degradation Human Socio-economical Systems Ecosystem Atmosphere

8. Reasoning across scales Atomic-molecular Macro-Scale Large-scale 3. Animal/People bodily functions 4. Dead body decay 5. Burning fossil fuels 2. Animal/People Growth 1. Plant Growth Organic-carbon oxidation— cellular respiration and combustion Organic-carbon transformation— digestion & biosynthesis Organic-carbon generation— photosynthesis Carbon Cycle Energy flow

9. Macroscopic Scale An example of Reasoning across scales: Human Breathing CO 2 O2O2 Atomic-molecular Scale O 2 + organic molecules  CO 2 + H 2 O Reasoning across scales: Patterns observed at the macroscopic scale may be different from processes at the atomic-molecular and large scalesReasoning across scales: Patterns observed at the macroscopic scale may be different from processes at the atomic-molecular and large scales

10. ASSESSMENT QUESTION 2 How do students reason about carbon- transforming processes? How shall we assess their understanding?

11. Students’ informal Explanation Force-dynamic Explanations: Actors, Enablers, Results Water Soil Sunlight Air

12. ASSESS STUDENT UNDERSTANDING: Process dimension Atomic-molecular Macro-Scale Large-scale 3. Animal/People bodily functions 4. Dead body decay 5. Burning fossil fuels 2. Animal/People Growth 1. Plant Growth Organic-carbon oxidation— cellular respiration and combustion Organic-carbon transformation— digestion & biosynthesis Organic-carbon generation— photosynthesis Carbon Cycle Energy flow The first dimension of the assessment: 6 Macroscopic Processes 6. Cross processes

13. ASSESSMENT QUESTION 3 How shall we analyze complex accounts from students?

14. Assess student understanding: Practice dimension Scale Macro-scaleMicro-scaleLarge-scale Matter Explaining Materials Practice: Explain changes of matter qualitatively Explaining Subsystems Practice: Explain macroscopic processes in terms of processes at a smaller scale; Explain macroscopic substance properties in terms of structure or properties at a smaller scale Explaining Large-scale systems Practice: Classify macroscopic processes and explain their connections in terms of patterns at large-scale. Classify macroscopic substances and explain their similarities. Explaining Mass Practice: Explain changes of matter quantitatively Energy Explaining Energy Practice: Explain changes of energy qualitatively BEAR Assessment System: Progress Variables; Levels of AchievementBEAR Assessment System: Progress Variables; Levels of Achievement Use a fine-grained measure to assess students’ understanding of the three scientific elements and two characteristic ways of reasoning.Use a fine-grained measure to assess students’ understanding of the three scientific elements and two characteristic ways of reasoning. Use matter conservation to constrain accounts about processes Use energy conservation and degradation to constrain accounts about processes Reasoning across Reasoning across scales The second dimension of the assessment: 5 Practices

15. Assess student understanding from two dimensions

16. Learning Progression Framework Achievement Levels Progress Variables Explaining Materials Explaining Mass Explaining Energy Explaining Subsystems Explaining Large-scale systems Level 4 Level 3 Level 2 Level 1

17. Explaining Materials Practice Level DescriptionExamples 4 Explaining about chemical identity of molecules and atom re- arrangement in chemical reaction Carbon dioxide reacts with water to produce glucose and oxygen is also released. 3 Explaining about changes involving atoms and molecules The sugar is energy. It turns into energy to help it [the tree] grow. 2 Explaining changes of the stuff that make up actors and enablers The wood turns into ashes and smoke which is released into air. 1 Explaining about actors and enablers You have to have a wick, or else it couldn’t burn on and it would be nothing for the fire to stay on. (Flame—Actor; Wick—Enabler that holds the flame)

18. Explaining Large-scale systems Practice Level DescriptionExamples 4 Carbon cycle & energy flow Light energy goes into the EcoSphere and heat is released out of the EcoSphere. 3 Carbon cycling without transformation; Energy cycling Explain how three things—algae, shrimp, and bacteria—stay alive in an EcoSphere. Energy cycles between shrimp and the algae it’s taking, like chemical energy from the algae and using it to live. Shrimp breathe, right. It would probably provide the oxygen for the algae to live so that would be like chemical energy I guess. 2 Stuff moving in food chain, gas cycle, etc. Explain the arrows in the diagram (loop diagram about interactions between human system, ecosystem and atmosphere) The boy is breathing out the carbon dioxide. The trees collect the carbon dioxide from the atmosphere and then produce oxygen, clean oxygen that we can breath. 1 Sequences of events; Perceptions Explain similarities and differences among tree growth, car running and flame burning The car has gas in it, so that it changes the air. You can smell the gas. You can smell trees. If you are buy like a maple tree candle, you would smell the maple sort of.

19. Additional Slides

20. An example of emergent property: Chemical energy Organic carbon-containing substances contain energy due to the configuration of atoms in molecules (C-C and C-H bonds); Carbon atoms do not contain energy.Organic carbon-containing substances contain energy due to the configuration of atoms in molecules (C-C and C-H bonds); Carbon atoms do not contain energy.

21. Consistency of students’ reasoning Consistency across the five practicesConsistency across the five practices Correlation coefficients range from.789 to.907Correlation coefficients range from.789 to.907 Consistency across different contexts (eight interview tasks)Consistency across different contexts (eight interview tasks) Correlation coefficients range from.690 to.908Correlation coefficients range from.690 to.908

22. Validity Analysis between interview data and written data Boxplots for Interview vs. Written Assessment

23. Validity Analysis between interview data and written data

25. Scatter plot of Average Interview Scores and Written Assessment Ability Estimates (r=.526)