Accelerating Future Possibilities for Assessment and Learning Technology-Enabled Measurement: Looking Back to Move Ahead Greg Chung UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Annual CRESST Conference, Los Angeles September 9, 2005

2 Organization of Talk Technology-Enabled Measurement Resolution Performance-Sensing Examples Circuit analyses Rifle marksmanship Automated Reasoning Data fusion and inferencing

3 Looking Back to Move Ahead Look back 30±10 years for potential technologies and methods – Internet (ARPA Net/ALOHA Net, 1972), relational DBs (IBM, 1970), sensing and telemetry (NASA, 1960s-70s), online monitoring (Dominick, 1973) – Constraint networks (Montanari, 1974), probabilistic causal reasoning (Suppes, 1970), knowledge representation (Minsky, 1975) – Adaptive/personalized instruction/ teaching machines (Glaser, Lumsdaine, Keller, s)

4 Technology-Enabled Measurement The use of technology to measure and assess human learning and performance Principal advantage is that technology- enabled measurement provides information about processes that cannot be obtained feasibly in any other way Order of magnitude increase in the scope, frequency, and resolution of what can be measured

5 Resolution The degree of detail that can be distinguished in students’ ongoing performance Expose black-box processes Process-oriented Fine-grained

6 Low Resolution

7 High Resolution We are here

8 Resolution How much do you really know about: How much someone knows (or doesn’t know) about something? What they can do (or can’t do)?

9 Examples Circuit Analyses Performance sensing: exposing students’ problem solving processes USMC Rifle Marksmanship Performance sensing: exposing shooters’ fine motor control processes

10 Higher Education Example Large-Class Instruction Instructor marches along with – Little direct feedback from students – Little or no information about what students know

11 How much do you really know whether students are getting it?

12

13 Instructor’s View

14 All students in session participated, drastically improved interaction Clear and immediate feedback Rate of receiving questions and observing responses to problems is much higher than conventional sessions Method exceeds interactivity of one-on-one office hour visit Instructor Observations

15 Example 2: USMC Rifle Marksmanship

16 USMC Rifle Marksmanship 500 yards

17 How much do you really know about these shooters?

18 Measuring Rifle Marksmanship Knowledge and Skills Rifle marksmanship is a combination of cognitive, affective, motor (gross and fine) skills, and uncontrollable equipment and environment variables Actual process of shooting is a black box—very difficult to observe fine-motor control processes – Practically impossible to observe all processes in parallel at the time the shot is taken

Why it Matters Soldier, Afghanistan 02 NY Times, 8/2/04 Marine, Iraq NY Times, 10/18/04 Marine, Haiti NY Times, 3/8/04

20 Performance Sensing

21 Trigger Control

22 Sensing Example Remarks Able to sense black-box processes that were previously unobservable—increase in resolution – Circuit analyses—inference engine in instructor’s head, working toward automating approach – Marksmanship—inferencing will be computer-based Highlights need for automated method for fusing data …

23 Automated Reasoning

24 Automated Reasoning Data Fusion Problem Given disparate data types, large quantity of data, and inherent uncertainty in the data – Need a way to reason about the data Bayesian networks useful as a way to model the phenomena – Model phenomena causally – Render probabilistic judgments about whether learner is in particular states

25 Circuit Analysis Bayesian Network

26 Towards Individualized Diagnosis and Prescription Model of Knowledge Dependencies and Performance Dependencies Recommender Content probabilities individualized feedback and remediation knowledge fine motor processes performance outcome

27 A Really Bold Assertion Technology Will Revolutionize the Field Broaden the way we think about how we measure human learning and performance Technology-enabled measurement affords increased resolving power “Through measurement to knowledge” (Onnes, 1882)