Scientifically Informed Digital Learning Interventions Financial and Intellectual Support: The William and Flora Hewlett Foundation The National Science Foundation A.W. Mellon Foundation Carnegie Mellon University One Example: The Open Learning Initiative at Carnegie Mellon

The Challenge To learn ways to design and build fully web-based courses which by rigorous assessments are proven to be as good or better than traditional teaching methods Why? –Increasing access –Improving effectiveness –Providing flexibility for faculty and students –Containing costs

A Flaw and an Opportunity Current structure of higher education presents substantial roadblocks to the application of proven results and methodologies from the learning sciences eLearning interventions, developed by teams rather than individuals, are more conducive to making the practice of education more scientific and effective

OLI Guiding Assumptions Digital learning interventions can make a significant difference learning outcomes Designs grounded in contemporary learning theory and scientific evaluation have the best chance of achieving that goal A possible, acceptable outcome of the OLI efforts is failure or mixed failures and successes – we are doing “action research,” not promoting eLearning for its own sake

OLI Guiding Assumptions Formative assessment will be a major feature (and a major component of the cost) of the designs and iterative improvements of the courses IT staff working with faculty is too limited a partnership – learning scientists, HCI experts, and assessment experts must be part of design, development, production and iterative improvement

Open Learning Initiative Courses Statistics Modern Biology Chemistry French Engineering Statics Causal and Statistical Reasoning Economics Logic and Proofs Physics Empirical Research Methods Computational Discrete Mathematics

Try it Yourself Don’t expect an “OCW experience”…this project has a different set of goals than OCW “Clicking around” will be unsatisfying: these interventions are designed to support a novice learner in acquiring knowledge working on their own

Key Elements in OLI Courses Theory Based: Course and individual lesson designs based on current theories in the learning sciences Feedback Loops: Courses record student activity for robust feedback mechanisms Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

Theory Based: Build on Prior/ Informal Knowledge

Theory Based: Provide Immediate Feedback in the Problem Solving Context

Theory Based: Promote Authenticity, Flexibility & Applicability Learning environments with ambiguous problems that require flexible application of procedural knowledge

Feedback Loops in Learning

Evaluation Chemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out. Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

Evaluation Statistics 1 st Study:

n Average % correct Pre Post Increase: 7.9% [t(487) = 13.8, p <.001] n Average % correct Pre Post Increase: 11.7% [t(23) = 4.7, p <.001]  CAOS Sample:  CMU OLI Course Sample: Evaluation

Measured learning Outcome % correct CAOS% correct CMU PrePostPrePost Box plots provide accurate estimates of % data above & below only for quartiles Correctly estimate and compare SD’s for different histograms Correlation does not imply causation Calculating appropriate conditional probabilities given table of data Evaluation

Accelerated Learning Study Taught Carnegie Mellon Introductory Statistics course in a blended mode (one in class meeting per week) in half a semester The OLI Statistics course was the “textbook” OLI course provided the professor immediate feedback on students’ performance We compared learning outcomes in the two different treatments

Accelerated Learning Study OLI students significantly outperformed Traditional “control” students on the CAOS post-test.

Accelerated Learning Study OLI students showed significantly greater gains (pre to post) than the Traditional “control” students on the CAOS test.

Student Satisfaction –End of course survey for online section: All students reported at an increase in their interest in statistics. 75% Definitely Recommend 25% Probably Recommend 0% Probably not Recommend 0% Definitely not Recommend

Feedback Loop – Current Research Instructors can use such data to adjust their teaching to students’ needs.

Learning Curve Analysis on Stoichiometry Data

The Vision – Digital Dashboard for Teaching and Learning: Instructor assigns students to work through online instruction System collects data as students work System automatically analyzes and organizes the data to present instructor with the students’ current “learning state” Instructor reviews this data summary and adapts instruction accordingly

The Anticipated Benefits Instructors get a window onto students’ progress They can adapt their teaching accordingly Students get better feedback to monitor and adjust their learning Strengthens the student-instructor connection

Core OLI Community Faculty Content Experts Learning Scientists Human Computer Interaction Software Engineers Evaluation/Assessment Specialists Learners A community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity” Herbert Simon (Candace Thille – Director)