1. Assessment and Learning Two Sides of the Same Coin.
Diana Bajzek, Associate Director Office of Technology for Education Carnegie Mellon University
Gordon Rule Professor Department of Biological Sciences
Additional authors:
Judy Brooks, William Jerome, Marsha Lovett, John Rinderle, Candace Thille

2. Outline Content Delivery & Analysis of student responses.
OLI – Online Learning Initiative
DDL – Digital Dashboard for learning
Simulator/Animation environment.
Development of authentic simulator environment
Embedding in OLI
Coupling of simulator environment to assessment.
Detached
Proximal
Associated
Tightly integrated
Reporting of student data by OLI/DDL.

3. Carnegie Mellon University Office of Technology for Learning
OLI: Research project creating exemplary online courses using the best information the learning sciences have to offer.
Working in courses which are part of Open Learning Initiative -OLI
Real courses, some fully online, some blended - all publicly available.
These courses are designed by teams of learning sciences researchers, technologists, and domain experts.
OLI courses provide rich interactive learning environments with comprehensive activity logging tools.
Digital Dashboard for Learning (DDL): Unique feedback tool for instructors and students.

4. The Course Design Triangle
Descriptions of what students should be able to do at the end of the course
Tasks that provide feedback on students’ knowledge and skills
Contexts and activities that foster students’ active engagement in learning
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies.
In all OLI courses, we begin our design by articulating a set of student- centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. As the course design triangle shows, designing effective courses requires the alignment of three major components: the learning objectives, the assessments, and the instructional activities. Student learning is enhanced when the instructor and the students have a clear sense of the learning objectives, the instructional activities provide practice and support for the acquisition of learning objectives, and the assessments provide students with opportunities to demonstrate and practice the skills articulated in the objectives
Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
The Course Design triangle representation is from the Eberly Center of Teaching Excellence, This representation is a distillation of the Dick and Carey Model (Dick, W. & Carey, L. The Systematic Design of Instruction 3rd. ed., copyright 1990, 1985, 1978, by Scott, Foresman & Co.) to the three fundamental components, highlighting the importance of the alignment of the elements and the iterative, non- sequential nature of the process.

5. The Course Design Triangle
Summative Assessments
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies.
In all OLI courses, we begin our design by articulating a set of student- centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. As the course design triangle shows, designing effective courses requires the alignment of three major components: the learning objectives, the assessments, and the instructional activities. Student learning is enhanced when the instructor and the students have a clear sense of the learning objectives, the instructional activities provide practice and support for the acquisition of learning objectives, and the assessments provide students with opportunities to demonstrate and practice the skills articulated in the objectives
Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
The Course Design triangle representation is from the Eberly Center of Teaching Excellence, This representation is a distillation of the Dick and Carey Model (Dick, W. & Carey, L. The Systematic Design of Instruction 3rd. ed., copyright 1990, 1985, 1978, by Scott, Foresman & Co.) to the three fundamental components, highlighting the importance of the alignment of the elements and the iterative, non- sequential nature of the process.

6. The Course Design Triangle
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies.
In all OLI courses, we begin our design by articulating a set of student- centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. As the course design triangle shows, designing effective courses requires the alignment of three major components: the learning objectives, the assessments, and the instructional activities. Student learning is enhanced when the instructor and the students have a clear sense of the learning objectives, the instructional activities provide practice and support for the acquisition of learning objectives, and the assessments provide students with opportunities to demonstrate and practice the skills articulated in the objectives
Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
The Course Design triangle representation is from the Eberly Center of Teaching Excellence, This representation is a distillation of the Dick and Carey Model (Dick, W. & Carey, L. The Systematic Design of Instruction 3rd. ed., copyright 1990, 1985, 1978, by Scott, Foresman & Co.) to the three fundamental components, highlighting the importance of the alignment of the elements and the iterative, non- sequential nature of the process.

7. The Course Design Triangle
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies.
In all OLI courses, we begin our design by articulating a set of student- centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. As the course design triangle shows, designing effective courses requires the alignment of three major components: the learning objectives, the assessments, and the instructional activities. Student learning is enhanced when the instructor and the students have a clear sense of the learning objectives, the instructional activities provide practice and support for the acquisition of learning objectives, and the assessments provide students with opportunities to demonstrate and practice the skills articulated in the objectives
Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
The Course Design triangle representation is from the Eberly Center of Teaching Excellence, This representation is a distillation of the Dick and Carey Model (Dick, W. & Carey, L. The Systematic Design of Instruction 3rd. ed., copyright 1990, 1985, 1978, by Scott, Foresman & Co.) to the three fundamental components, highlighting the importance of the alignment of the elements and the iterative, non- sequential nature of the process.

8. The Course Design Triangle
Formative Assessments
We begin an OLI course development project by studying the teaching and learning challenges in the domain under development. Our studies include literature reviews, reviews of existing artifacts of student learning, classroom observations, lab studies and/or classroom-based studies.
In all OLI courses, we begin our design by articulating a set of student- centered measurable learning objectives and we design the instructional material to support students to achieve the articulated objectives. As the course design triangle shows, designing effective courses requires the alignment of three major components: the learning objectives, the assessments, and the instructional activities. Student learning is enhanced when the instructor and the students have a clear sense of the learning objectives, the instructional activities provide practice and support for the acquisition of learning objectives, and the assessments provide students with opportunities to demonstrate and practice the skills articulated in the objectives
Our instructional activities include small amounts of explanatory text and many simulations which capitalize on the computer’s capability to promote interaction and to display digital images and video. Many of the courses including formal logic, statistical reasoning and chemistry include virtual labs that promote flexible and authentic exploration and problem solving. A hallmark of all OLI courses is the frequent opportunities we provide students to assess their own learning and receive context-specific and targeted feedback on their work.
The Course Design triangle representation is from the Eberly Center of Teaching Excellence, This representation is a distillation of the Dick and Carey Model (Dick, W. & Carey, L. The Systematic Design of Instruction 3rd. ed., copyright 1990, 1985, 1978, by Scott, Foresman & Co.) to the three fundamental components, highlighting the importance of the alignment of the elements and the iterative, non- sequential nature of the process.

9. Feedback Loops in Instruction/Assessment
Instructors and Students Would Like To:
Identify difficult concepts
Detect common misconceptions
Reveal murkiest points
Assess the depth of understanding of topics
Feedback Loops in Learning
Assessments
Active Learning
Instructors can use such data to adjust their teaching to students’ needs.
This feedback loop is always cycling in traditional face-to-face classes and it works well there when you have an expert teacher and a not unreasonably heterogenous class: the instructor sees students’ faces look confused or sees poor results on a test and can adjust instruction to help.
But this cycle is also something we can leverage in non-traditional/online instruction: OLI courses that are used in a hybrid model. OLI and PSLC courses are well instrumented so data are there, and we have new techniques for automating complex data-analysis procedures that would be necessary to get the data in a meaningful form.
Instructional Improvement

10. Modification of Instruction by incorporating feedback.

11. Modification of Instruction by incorporating feedback.

12. Design of Animations/Simulations
Glucose Transport into Liver
Select concept.
Static image.
Movie clip.
Simulation
Student Interaction
Outside
Glucose
Cell Membrane
Glucose Transporter
Hexose Kinase
(G → G-P)
Inside

13. Design of Animations/Simulations
Simulator Environment:
Cast are given xi, vi.
Follow Newton’s laws & instructor defined rules.
Glucose Transport into Liver
Outside
Glucose: Can’t cross the membrane, unless it encounters its transporter.
Membrane: Nothing goes through, in this example.
Glucose Transporter: Only moves glucose across the membrane
Inside
Hexokinase: Enzyme that only binds glucose. Bound glucose converted to G-P
Hexose_kinase.html

14. Design of Animations/Simulations
Enzyme Kinetics
Select concept.
Static image.
Movie clip.
Simulation
Student Interaction
1. Simulator
Stage
3. Data Output
2. Student Input

15. Providing Context for Simulations:

16. Disconnected Assessment:

17. Proximal Assessment

18. Associated Assessments

19. Coupled Assessments Context given within assessment.
Student prompted to interact with the simulator to gain information to answer question.
Student input passed to simulator, changing its state.
Simulator can be used to drive data output.
Question posed:
Hints available
Individual feedback for answers.
Student response & assessment activity can control simulator environment.
Control of flow is possible as the student progresses through exercise.
Inhibitors_simulator.html

20. Coupled Assessments Context given within assessment.
Student prompted to interact with the simulator to gain information to answer question.
Student input passed to simulator, changing its state.
Simulator can be used to drive data output.
Question posed:
Hints available
Individual feedback for answers.
Student response & assessment activity can control simulator environment.
Control of flow is possible as the student progresses through exercise.
tca_simulator.html

21. Assessment System Familiar question types
Sophisticated evaluation (scoring)
Flexibly evaluated according to criteria specified by the instructor
Multi-part questions

22. Feedback Report for the assessment as seen by an instructor
Class Participation
Best and Worst
Questions
Individual Question

23. Feedback Report for the assessment as seen by an instructor
Class Participation
Best and Worst
Questions
Individual Question

24. Feedback Report for the assessment as seen by an instructor
Class Participation
Best and Worst
Questions
Individual Question

25. OLI project: http://www.cmu.edu/oli/ Biology project:
Funding:
The William and Flora Hewlett Foundation
Howard Hughes Medical Institute