1. UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing V4, 1/18/07 Research on Adult Learning Advanced Technology Applications to Assessment of Complex Cognitive Skills CRESST Conference January 22, 2007 Bill Bewley UCLA/CRESST

2. 1/21 The Elevator Speech Information technologies can be used to assess complex knowledge and skills CRESST has conducted research on applications of advanced technologies to assessment of planning and problem- solving in military tasks The approaches can be applied to assessment of complex knowledge and skill in other domains

3. 2/21 Approaches Simulation-based assessment The product The process and product Situational judgment Sensor-based assessment Modeling and interpretation

4. 3/21 Assessing the Product Link Architecture Planning

5. 4/21 The Process and the Product Click to check threat information Click to view threat/defense geometry Place assets, location and role assignment Adjust defense geometry Air Defense Planning

6. 5/21 Known Distance Coaching The Process and the Product Diagnosis

7. 6/21 The Decision Analysis Tool The Process and the Product Create options Adjust parameters

8. 7/21 The Process and the Product Decision Analysis Tool results Clickstream analysis shows different strategies for each group Gaming: focus on one parameter at a time (utility, probability, threshold), end by adjusting threshold to turn light green for favored option Coarse: make big changes in utility, few changes in probability Holistic: a variety of changes, not focusing on one parameter at a time and not gaming But small N, so need additional data collection

9. 8/21 Situational Judgment Evaluation of Shooting Positions

10. 9/21 Situational Judgment As a coach, observe shooters on the firing line Combat Marksmanship Coaching

11. 10/21 What kind of error did you observe? Sequence Skipped Step Wrong Execution >> Choose all that apply << Situational Judgment Fault check and describe what’s wrong, or fix it by dragging and dropping to fill in missing step(s) correct the execution of a step correct the sequence of steps Combat Marksmanship Coaching

12. 11/21 Sensor-Based Assessment Pressure sensor on trigger Eye tracker Motion sensor on muzzleTrigger squeeze data Laser strike sensor Eye position data Rifle Marksmanship

13. 12/21 Modeling and Interpretation Modeling using ontologies Diagnosis using Bayes nets Artificial neural nets (current research) Hidden Markov models (current research) Prescription based on diagnosis and domain ontologies

14. 13/21 Modeling and Interpretation Domain knowledge: rifle marksmanship

15. 14/21 Modeling and Interpretation A domain ontology: rifle marksmanship Concepts and relations between concepts, in a database

16. 15/21 Modeling and Interpretation Example 1: Marksmanship concepts Represent the domain using an ontology Estimate what the learner knows about a domain given performance data on assessments Use Bayesian networks to fuse assessment data and infer understanding of domain and topics within domain

17. 16/21 Modeling and Interpretation Example 1: Marksmanship concepts Bayesian Network Model of Knowledge Dependencies Recommender Ontology of Marksmanship Domain content item-level scores probability of knowing/ not knowing content individualized feedback and content

18. 17/21 Modeling and Interpretation Example 1: Marksmanship concepts

19. 18/21 Modeling and Interpretation Example 1: Marksmanship concepts Results Prescribed training appears to have increased conceptual knowledge Performance on targeted concepts improved in the experimental group No change in the control group Bayesian network appears to be capturing the knowledge dependencies Similar to Marines’ self-assessments Correlated with shooting scores

20. 19/21 Modeling and Interpretation Example 2: Bayes nets for Air Defense Use the probabilities that knowledge is in low, medium, or high categories for each measure to predict prior knowledge

21. 20/21 Modeling and Interpretation Example 2: Bayes nets for Air Defense Results 30% of students were identified as having sufficient prior knowledge to skip air defense instruction—surprisingly high Explanation: instructors found that due to a course redesign at another school, students had air defense instruction just before entering the SWOS course

22. 21/21 ©2007 Regents of the University of California