1. Scientifically Informed Digital Learning Interventions
One Example: The Open Learning Initiative at Carnegie Mellon
Financial and Intellectual Support:
The William and Flora Hewlett Foundation
The National Science Foundation
A.W. Mellon Foundation
Carnegie Mellon University
CHANGE

2. The Challenge
To learn ways to design and build fully web-based courses which by rigorous assessments are proven to be as good or better than traditional teaching methods
Why?
Increasing access
Improving effectiveness
Providing flexibility for faculty and students
Containing costs

3. A Flaw and an Opportunity
Current structure of higher education presents substantial roadblocks to the application of proven results and methodologies from the learning sciences
eLearning interventions, developed by teams rather than individuals, are more conducive to making the practice of education more scientific and effective

4. OLI Guiding Assumptions
Digital learning interventions can make a significant difference learning outcomes
Designs grounded in contemporary learning theory and scientific evaluation have the best chance of achieving that goal
A possible, acceptable outcome of the OLI efforts is failure or mixed failures and successes – we are doing “action research,” not promoting eLearning for its own sake

5. OLI Guiding Assumptions
Formative assessment will be a major feature (and a major component of the cost) of the designs and iterative improvements of the courses
IT staff working with faculty is too limited a partnership – learning scientists, HCI experts, and assessment experts must be part of design, development, production and iterative improvement

6. Open Learning Initiative Courses
Statistics
Modern Biology
Chemistry
French
Engineering Statics
Causal and Statistical Reasoning
Economics
Logic and Proofs
Physics
Empirical Research Methods
Computational Discrete Mathematics
OLI and Pittsburgh Sciences of Learning Center (PSLC) share the vision of improving education by transferring knowledge developed in learning science research into teaching and learning practices. The PSLC and OLI are both research and development projects, but we very have different emphasis on the two activities. The PSLC is a capital “R” developing/testing/confirming learning science Research. It is development with a small “d” – development only as needed, to support the research. OLI, in contrast, is the capital “D” with it’s major focus on developing courses and translating and applying theoretical findings to concrete educational settings with a small “r”, as needed, to support the course development. The PSLC was created and funded by the National Science Foundation a couple of years after the inception of the OLI. It’s hardly surprising that both exist at Carnegie Mellon, given our institutional strengths in cognitive science. But the happy circumstance of the creation of the PSLC as the OLI has grown has provided unexpected synergies.
The Open Learning Initiative (OLI) is a project devoted to developing scientifically based, openly available online courses and course materials. OLI started in the fall of 2002, funded by a grant from The William and Flora Hewlett Foundation. The project has the potential to have a positive impact on higher education by increasing access, enhancing quality and providing new exemplars for online courses and course materials. In this presentation will describe how we have made use of experts in and information from contemporary cognitive and learning sciences to produce high quality online courses and course materials and we will show demonstrations of those courses.

7. Try it Yourself http://www.cmu.edu/oli
Don’t expect an “OCW experience”…this project has a different set of goals than OCW
“Clicking around” will be unsatisfying: these interventions are designed to support a novice learner in acquiring knowledge working on their own

8. Key Elements in OLI Courses
Theory Based: Course and individual lesson designs based on current theories in the learning sciences
Feedback Loops: Courses record student activity for robust feedback mechanisms
Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers
CHANGED

9. Theory Based: Build on Prior/ Informal Knowledge
In the economics course we use the affordances of the web to start with students' informal experiential understanding and bridge to more powerful formal forms of understanding. Students begin each economics unit by participating in a carefully structured online experiment with other remote students. In each online experiment the student is an active participant attempting to make deals with other traders in a market. Each student is randomly assigned a role and cost price in each experiment. To make a profit in the experiment, the students must either sell higher than their cost price or buy lower then their cost price. No one knows each others role or cost price. When we use this material at CMU, part of the students grade is dependent on how much profit each student makes in each of the experiments. The students experience first hand the issues that all economic agents confront.
Once the experiment is complete, the data generated by the students’ participation in the experiment is transferred to a custom workbook. The students can now see the details of the experiment, e.g. the number of suppliers and the number of demanders and everyone’s cost price. They use this data to complete the tables in the workbook and see how well the economic theories predict their behavior. This approach of bridging from experiential knowledge to formal theory allows students to learn very sophisticated concepts in economics such as the theory of Asymmetric Information and Adverse Selection and how to apply these ideas to real-world markets and understand various policy options.
Mini tutors are used throughout OLI courses in conjunction with other learning activities. In our economics course, for example, we use the mini tutors to support students in learning basic skills of drawing and interpreting supply and demand graphs.
THIS AND NEXT SLIDE ARE ALSO DESIGNED FROM RESEARCH ON WHAT PROMOTES DEEP LEARNING: E.G., BUILDING ON KNOWLEDGE AND TIMELY FEEDBACK. I THINK IT’S THERE, BUT YOU MIGHT WANT TO BE EVEN MORE EXPLICIT ABOUT THAT POINT.

10. Theory Based: Provide Immediate Feedback in the Problem Solving Context
The MiniTutors are grounded in studies that have attributed the sizeable learning gains that students achieve with human tutors to the feedback the tutor gives in the problem solving context. The hints and feedback change depending on which part of the exercise the student is attempting. Notice that the hints are given in three levels with the first level of hint orienting the student in general terms, the second level of hint restating the rules or strategies that the student should apply in solving the problem, and the final level of hint, or “bottom out hint” walking the student through the solution for that step in the process. This demonstrates the methodology of a cognitive tutor: making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.
This tutor is in a section of the Statics Course on Summing Force Vectors, and helps students learn how to determine the sum of concurrent forces by resolving them into components. It is intended to be an opportunity for students to do a "self-check" to make sure they understand the concept. However, if the student is unsure of the procedure for solving the problem, the first hint provides a link which, when clicked, expands the tutor into the various steps needed to solve the problem.
In addition, the problem statement for this tutor includes randomly-generated values for the three forces, so the student can work through the tutor multiple times, receiving a different problem each time, until the student is confident that he or she understands the concept.

11. Theory Based: Promote Authenticity, Flexibility & Applicability
Learning environments with ambiguous problems that require flexible application of procedural knowledge
From mathematical procedures to chemical phenomena (use in chemistry) - Virtual laboratory
From chemical phenomena to real world (transfer to real world) - Scenario based learning
Rather than teaching abstract skills out of context, the chemistry course situates the learning in an authentic investigation that addresses significant questions Our unit on stoichiometry is situated in a real world problem of arsenic contamination of the water supply in bengladesh. The virtual lab provides opportunities for students to interact with the environment by exploring and manipulating objects, wrestling with questions and performing experiments. Chemistry problems are approached as a chemist would approach them to solve real world problems This approach promotes deeper learning and lets different students solve problems in different ways.
Flexible Application of Knowledge:
Here, students are given a goal and of the lab is configured with various chemical
solutions, equipment and solution viewers. This is similar to setting up a physical lab and many of our
activities are indeed parallel to what would be done in physical lab. However, since experiments can
be done quickly and safely, students can be given greater flexibility in the design of the experiment.
This, combined with the ability to look directly inside a solution to see the types of species and their
concentrations, leads to entirely new types of activities that would not be feasible in a physical
laboratory. For instance, in our “Oracle” problem, students must determine the reaction between some
fictional chemical species including stoichiometric coefficients (i.e. A +2C 􀃆 B + D). This can be
done quickly because the concentrations of all species are always present. This requires a flexible
application of their knowledge of stoichiometry and limiting reagents and has revealed some
interesting issues regarding students understanding of chemical reactions [Yaron et al, 2003].
Help students move beyone shallow problem solving strategies:
A a useful but potentially superficial strategy for word
problems is to categorize the given and requested information and then find equations that relate the
information. For instance, a calorimetry text problem may give a measured change in temperature (∆T
given) and ask for a heat (q requested), for which as student may identify q = m Cp ∆T as an
appropriate equation. An activity that requires design of an experiment to measure the heat of a
process requires deeper reflection, since the student must realize that this equation represents an
experiment in which a temperature change is used to measure heat. Our observations show that
students find the experimental design problem considerably more difficult than the text problem suggesting that this connection between equation and physical process does involve additional learning.

12. Feedback Loops in Learning
Use available data on student learning to improve instruction
Note that these different data sources have different grain sizes, time scales, etc.

13. Evaluation
Chemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out.
Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out
CHANGED

14. Evaluation
Statistics 1st Study:
Changed

15. Evaluation Increase: 7.9% [t(487) = 13.8, p <.001] Increase: 11.7%
CAOS Sample: n
Average % correct
Pre
488
43.3
Post
51.2
Increase: 7.9%
[t(487) = 13.8, p <.001]
CMU OLI Course Sample: n
Average % correct
Pre 24
55.8
Post
66.5
Increase: 11.7%
[t(23) = 4.7, p <.001]

16. Evaluation
Measured learning Outcome
% correct CAOS
% correct CMU
Pre
Post
Box plots provide accurate estimates of % data above & below only for quartiles
22.2
50.0
Correctly estimate and compare SD’s for different histograms.
31.5
46.4
66.7
83.3
41.8
59.3
75.0
Correlation does not imply causation
51.9
49.4
48.1
70.8
Calculating appropriate conditional probabilities given table of data
49.6
47.4
70.4
CHANGED

17. Accelerated Learning Study
Taught Carnegie Mellon Introductory Statistics course in a blended mode (one in class meeting per week) in half a semester
The OLI Statistics course was the “textbook”
OLI course provided the professor immediate feedback on students’ performance
We compared learning outcomes in the two different treatments

18. Accelerated Learning Study
OLI students significantly outperformed Traditional “control” students on the CAOS post-test.

19. Accelerated Learning Study
OLI students showed significantly greater gains (pre to post) than the Traditional “control” students on the CAOS test.

20. Student Satisfaction End of course survey for online section:
All students reported at an increase in their interest in statistics.
75% Definitely Recommend
25% Probably Recommend
0% Probably not Recommend
0% Definitely not Recommend
CHANGED

21. Feedback Loop – Current Research
Instructors can use such data to adjust their teaching to students’ needs.
This feedback loop is always cycling in traditional classes and it works well there when you have an expert teacher and a not unreasonably heterogenous class: the instructor sees students’ faces look confused or sees poor results on a test and can adjust instruction to help.
But this cycle is also something we can leverage in non-traditional/online instruction: OLI courses that are used in a hybrid model -- and indeed in many online learning environments offer students online materials/activities and the instructor’s support: instructors want to be able to adapt the instruction to their students, on-line learning environments (OLI and PSLC courses but others as well) are often well instrumented so data are there, and we have new techniques for automating complex data-analysis procedures that would be necessary to get the data in a meaningful form.
Make Eberly connection…

22. Learning Curve Analysis on Stoichiometry Data
This is a learning curve graph of the knowledge components in the stoichiometry tutor (OLI Chemistry Course). The Y-axis is the “Assistance Score” (the number of hints the students requests and the number of incorrect answers they select) for the selected knowledge component and the X-axis is the opportunity number. The curve is trending downward which means that the students needed fewer hints and gave fewer incorrect answers as they progressed through the material; shows that learning is occurring. This is a graph for all knowledge components together. The tool also allows us to look at graphs for each individual knowledge component and to identify knowledge components such as “set denominator value of Avagadro’s number” which may show a learning curve that is not trending so neatly downward and may indicate a need for revision of the teaching approach.

23. The Vision – Digital Dashboard for Teaching and Learning:
Instructor assigns students to work through online instruction
System collects data as students work
System automatically analyzes and organizes the data to present instructor with the students’ current “learning state”
Instructor reviews this data summary and adapts instruction accordingly
CHANGED

24. The Anticipated Benefits
Instructors get a window onto students’ progress
They can adapt their teaching accordingly
Students get better feedback to monitor and adjust their learning
Strengthens the student-instructor connection
Beyond ITS, having a human involved and informed gives you the best of both worlds
Mention statistics example: spending so much time on boxplot basics when students could get it very easily
Mention Biology course and Bill’s experience

25. Core OLI Community Faculty Content Experts Learning Scientists
Human Computer Interaction
Software Engineers
Evaluation/Assessment Specialists
Learners
A community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

26. (Candace Thille – Director)
“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”
Herbert Simon
(Candace Thille – Director)