1. A Value-Based Approach for Quantifying Scientific Problem Solving Effectiveness Within and Across Educational Systems Ron Stevens, Ph.D. IMMEX Project UCLA School of Medicine Vandana Thadani, Ph.D. Loyola Marymount University

2. The Challenging Question: What is a suitable description of problem solving that can capture important cognitive and performance information about an individual’s problem solving, yet provide rapid and meaningful comparisons within and across science domains and educational systems? What is a suitable description of problem solving that can capture important cognitive and performance information about an individual’s problem solving, yet provide rapid and meaningful comparisons within and across science domains and educational systems?

3. Why Would Such a Measure(s) Be Useful? Could document the development of problem solving skills: Could document the development of problem solving skills: Throughout the year. Throughout the year. Across science domains. Across science domains. Would allow comparisons: Would allow comparisons: Across students classrooms, teachers and help guide professional development. Across students classrooms, teachers and help guide professional development. Across schools and school systems. Across schools and school systems.

4. Important Properties for a Generalizable Measure(s) Face, Construct, Concurrent and Divergent validity Face, Construct, Concurrent and Divergent validity Reliability Reliability Scalability Scalability Understandability Understandability Adaptability / Extensibility Adaptability / Extensibility

5. Theoretical Groundings: Strategy and Skill Development Each individual selects the best strategy for them on a particular problem. Each individual selects the best strategy for them on a particular problem. People adapt strategies to changing rates of success. People adapt strategies to changing rates of success. Paths of strategy development emerge as students gain experience; and, Paths of strategy development emerge as students gain experience; and, Improvement in performance is accompanied by an increase in speed and reduction in the data processed. Improvement in performance is accompanied by an increase in speed and reduction in the data processed.

6. A Framework for Assessing Problem Solving Skills How well / rapidly were the problems solved? ( easy to assess, but little strategic information ) How well / rapidly were the problems solved? ( easy to assess, but little strategic information ) Can hard / easy problems be solved? ( more difficult to assess, IRT estimates are useful ) Can hard / easy problems be solved? ( more difficult to assess, IRT estimates are useful ) What problem solving strategy was used? ( more difficult to assess ) What problem solving strategy was used? ( more difficult to assess ) Are the problem solving strategies improving with practice? ( more difficult to assess ) Are the problem solving strategies improving with practice? ( more difficult to assess ) What strategy will the student next use? ( hard to assess ) What strategy will the student next use? ( hard to assess ) …and the ability to generalize across domains and educational systems. …and the ability to generalize across domains and educational systems.

7. Problem-Solving With IMMEX tm IMMEX is a web-based learning system for promoting problem solving and decision making skills. IMMEX is a web-based learning system for promoting problem solving and decision making skills. The IMMEX system includes real-time student modeling capabilities for assessing and reporting student progress. The IMMEX system includes real-time student modeling capabilities for assessing and reporting student progress.

8. Hazmat: A Hazardous Materials Simulation Face Validity – The Tasks The Hazmat task is to analyze a toxic spill by using multiple chemical and physical tests. http://www.immex.ucla.edu/docs/ publications/pdf/its_paper.pdf http://www.immex.ucla.edu/docs/ publications/pdf/its_paper.pdf Stevens, R., Soller, A., Cooper, M., and Sprang, M. (2004). Intelligent Tutoring Systems. Lester, Vicari, & Paraguaca (Eds). Springer-Verlag Berlin Heidelberg, Germany. 7th International Conference Proceedings (pp. 580-591). Intelligent Tutoring Systems. Lester, Vicari, & Paraguaca (Eds). Springer-Verlag Berlin Heidelberg, Germany. 7th International Conference Proceedings (pp. 580-591).

9. Layers of Assessment Tools for Investigating Skill Development Student Ability Estimates First builds a model of the difficulties of each case based on student performance. Then each student is evaluated against this model. Strategy Models Self-organizing neural networks cluster similar performances into strategic topology maps. Progress Models Probabilistic models of sequences of neural network strategies.

10. Problem Sets Contain Multiple Cases of Varying Difficulty As expected, flame test negative compounds are more difficult than positive ones. The student abilities follow a normal distribution. This data is useful for comparing problem solving ability with other student assessments like standardized tests. It does NOT indicate HOW the problem was solved.

11. Defining Strategies with Artificial Neural Networks – The Idea Postulate a number of major strategies that can be applied to the problem….we often use 36. Postulate a number of major strategies that can be applied to the problem….we often use 36. Train the neural network with thousands of performances from students of many abilities using the tests they selected as input data. Train the neural network with thousands of performances from students of many abilities using the tests they selected as input data. The performances compete with each other for each neural network node such that those most similar are clustered together on the map. The performances compete with each other for each neural network node such that those most similar are clustered together on the map.

12. Defining Strategies with Artificial Neural Networks – The Data http://www.immex.ucla.edu/docs/publications/anndistinguish.htm Journal of the American Medical Informatics Association 3: 131-8, 1996. Stevens, R.; Lopo, A.; Wang, P.

13. Developing Progress Models From Sequences of Neural Network Strategies – The Idea Postulate a number of states that represent transitions students may pass through (often 3-5). Postulate a number of states that represent transitions students may pass through (often 3-5). Train Hidden Markov Models with many strategy sequences. Train Hidden Markov Models with many strategy sequences. Use the transition and emission matrices from the modeling to develop learning progress models. Use the transition and emission matrices from the modeling to develop learning progress models.

14. Developing Progress Models From Sequences of Neural Network Strategies –Examples

15. Developing Progress Models From Sequences of Neural Network Strategies –The Data State 1 55% Solved State 2 60% Solved State 3 45% Solved State 4 54% Solved State 5 70% Solved

16. Student Problem Solving Strategies Stabilize Rapidly

17. Collaborative Grouping Accelerates Strategy Adoption

18. The Big Problem for Dissemination ANN and HMM modeling are very useful research tools……but ANN and HMM modeling are very useful research tools……but Each problem set has its own ANN topology and state transitions. Each problem set has its own ANN topology and state transitions. So a teacher implementing a dozen IMMEX problem sets would need to understand 24 models! So a teacher implementing a dozen IMMEX problem sets would need to understand 24 models!

19. One Solution: A Value-based Approach Explore expressing problem solving as a value relating the efficiency of the process to the effectiveness of the outcomes. Explore expressing problem solving as a value relating the efficiency of the process to the effectiveness of the outcomes.

20. Strategic Efficiency Students demonstrating high strategic efficiency will make the most effective problem solving decisions using the least number of resources available. Resources can be costs, risks, time, etc. Students demonstrating high strategic efficiency will make the most effective problem solving decisions using the least number of resources available. Resources can be costs, risks, time, etc. A quality measure is also needed as not all resources will be equally applicable to the problem at hand. A quality measure is also needed as not all resources will be equally applicable to the problem at hand.

21. Start with ANN Defined Strategies That Differ in the Data Selected and Solve Rates

22. Efficiency Index: Outcomes Obtained vs. Resources Used For example, for one strategy 9 of the 22 items were selected by the majority of the students and with a solve rate of 1.33 the EI = 3.25 For example, for one strategy 9 of the 22 items were selected by the majority of the students and with a solve rate of 1.33 the EI = 3.25

23. Relating the EI to Strategies and Problem Difficulty

24. Steps for Calculating a Strategic Efficiency Index (EI) 1. Train ANN and calculate the EI for each node. 2. For each new performance, determine the matching strategy from ANN, and assign the associated nodal EI. 3. Determine if the case was solved or not. 4. Repeat for additional cases and average EI and solved rate 5. Calculate Quadrant Value (QV)

25. Calculate QV from Nodal EI and Outcome A performance at a particular node can either under perform or over perform the nodal average

26. Final Wrinkle--Problem Sets Vary in Difficulty and EI

27. The Solution: Normalize Performances to Quadrants Defined by the Mean EI and Solved Rates

28. The Classes of Individual Teachers Often Show Similar QV Averages Individual student problem solving progress from classes of four teachers. Reliability – Across Classroom Performances

29. QV Changes With Student Experience Construct Validity – Strategic Changes With Experience

30. Concurrent Validity: Correlations of IRT Problem Solving Scores and QV Indices with Achievement Scores

31. Are Students Problem Solving to Their Abilities? … it may depend on their teacher.

32. Correlations Between QV and Achievement Test Scores is Independent of Content Domain

33. Summary Detailed machine learning models of problem solving progress can be developed. Detailed machine learning models of problem solving progress can be developed. By focusing on the value of the problem solving process these models can be generalized and aggregated across content domains. By focusing on the value of the problem solving process these models can be generalized and aggregated across content domains.