1. The Effect of Generation on Robust Learning
Robert G.M. Hausmann &
Kurt VanLehn
2/18/2019
Pittsburgh Science of Learning Center

2. Placing Study within Robust Learning Framework
Primary research cluster
Interactive communication, coordinative learning, refinement & fluency
Independent variable(s)
IC: Reflective dialog, scripting collaboration, peer tutoring, peer observation of tutoring, …
CL: self-explanation vs. paraphrase, complete vs. incomplete worked-out examples, …
R&F: Feature focusing, example comparison, cognitive mastery, optimal spacing, …
Dependent variables used
Normal post-test, long term retention, transfer, future learning
2/18/2019
Pittsburgh Science of Learning Center

3. Pittsburgh Science of Learning Center
Outline
Next
Introduction
Objectives
Background
Research problem
Study
Interpretation
2/18/2019
Pittsburgh Science of Learning Center

4. Pittsburgh Science of Learning Center
Background
Self-explanation is a sense-making process that generates inferences to adapt material to background knowledge (Chi, 2000)
Passive
Active
Interactive
Explicit (general)
Description
Self-explanation
Self-explanation with feedback
Implicit (instance)
Example
Practice
Tutored practice
2/18/2019
Pittsburgh Science of Learning Center

5. Pittsburgh Science of Learning Center
Background
Self-explanation is a sense-making process that generates inferences to adapt material to background knowledge (Chi, 2000)
Paraphrasing aids the acquisition of shallow knowledge (Hausmann & Chi, 2002; VanLehn & Jones, 1995)
Passive
Active
Interactive
Explicit (general)
Description
Paraphrasing
Self-explanation with feedback
Implicit (instance)
Example
Practice
Tutored practice
2/18/2019
Pittsburgh Science of Learning Center

6. Pittsburgh Science of Learning Center
Research problem
Problem
Does self-explanation work in the classroom, even when controlling for active learning processes?
Hypothesis
Coverage Hypothesis
If the information is presented, and the student “actively” processes it, then they will exhibit robust learning.
Generation Hypothesis
If the information is generated by the student, then they will exhibit robust learning.
2/18/2019
Pittsburgh Science of Learning Center

7. Pittsburgh Science of Learning Center
Outline
Introduction
Objectives
Study
Method: conditions
Method: procedure
Results
Interpretation
Next
2/18/2019
Pittsburgh Science of Learning Center

8. Treatment: Self-explanation by reference
Design
Participants
Physics LearnLab
USNA: Physics II (N=104)
Materials
Study instructions
Video solutions to study
Andes homework system
Chapter exam (1 question)
Homework
Study domain: electrodynamics
AFL domain: magnetism
Para-phrase
Self-explain
Complete Example
n = 26
n = 27
Incomplete Example
n = 23
n = 28
Attrition: originally had access to 113 mids, 7 ~consent, 2 absent = 104 total
Deviation: did not follow directions by opening out of order and not watching the videos.
However, authentic classroom—students avail themselves of the instructional materials. Need the gating procedure due out this fall in OLI.
Incomplete examples: created by deleting the justification for the application of a knowledge component.
2/18/2019
Pittsburgh Science of Learning Center

9. Independent Variable: Study Strategy Instructions
Before students watched the videos, they received the following instructions:
We would like you to watch and listen to each step and then explain what it means to you/paraphrase in your own words. That is:
Self-Explaining Instructions
What new information does each step provide for you?
How does it relate to what you've already seen?
Does it give you a new insight into your understanding of how to solve the problems?
Does it raise a question in your mind?
Paraphrasing Instructions
How can I put this in my own words?
How can I say this in a different way?
In other words: _________________.
What's another way that I could put this?
Mention microphone & not a computer-based intervention
Spoken self-explanations:
SE are more natural when spoken in natural language than typed
More social; whereas, writing is more private
Note:
An example of a self-explaining episode was presented next. & the bullet list of prompts was found at the bottom of each video example.
You can see the explanation instructions emphasize new information, fitting it to current understanding and prior knowledge.
Paraphrase may self-explain. So getting the effect may be difficult.
Tell us whatever is going through your mind - even if it seems unimportant.
2/18/2019
Pittsburgh Science of Learning Center

10. Assistance Score = (hints+errors)/steps
What’s Wrong Help
Next Step Hint
Error (flag feedback)
Dependent Variable:
Assistance Score = (hints+errors)/steps
Bottom-out
hint
2/18/2019
Pittsburgh Science of Learning Center

11. Pittsburgh Science of Learning Center
Method: Procedure
Self-explain
Complete
Incomplete
Paraphrase
Example1
Self-explain
Complete
Incomplete
Paraphrase
Example2
Self-explain
Complete
Incomplete
Paraphrase
Example3
Warm-up Problem
Problem1
Problem2
Problem3
Study duration: 110 minutes
The manipulation is on the type of studying of the examples.
The problem-solving practice is on the same KCs studied in the examples.
2/18/2019
Pittsburgh Science of Learning Center

12. Summary of study design
Independent variables:
Study Strategy: self-explain or paraphrase
Example Type: complete or incomplete
Control for active learning!
Dependent variables:
Normal post-test: interspersed problem solving
Retention: non-isomorphic chapter exam problem
Transfer test: homework problems (electrodynamics)
Acc. Future Learning (AFL): homework problem (magnetism)
2/18/2019
Pittsburgh Science of Learning Center

13. Results: Normal Post-test
Isomorphic problem solving during laboratory period (electrodynamics)
ME: Study Strategy (paraphrase > self-explain)
Assistance Score: Normalized by dividing by the number of entries.
Repeated measures anova
DV: Assistance Score/steps F(1, 73) = 6.189, p < ME: talk eta^2: .078
2/18/2019
Pittsburgh Science of Learning Center

14. Results: Retention
Non-isomorphic exam problem one week later (electrodynamics)
Data source: 2/5 sections (n=46), scored by the *instructor*
Compliance is a limitation of LearnLab. (need more incentives for schools and instructors)
Medium duration: after the study, 6 days after the lab (i.e., normal test).
The findings in the classroom fail to replicate retention measures from laboratory studies
assessment may not be sensitive enough (i.e., one question without a log file).
Or the sample size is not large enough to detect a difference.
The complete self-explanation group was marginally higher than the complete paraphrase condition (LSD, p = .064).
2/18/2019
Pittsburgh Science of Learning Center

15. Results: Acc. Future Learning
Andes homework performance six weeks later (magnetism)
ME: Study Strategy (paraphrase > self-explain)
ANOVA (mag4a)
DV: Assistance Score/steps F(1, 46) = 5.223, p < Int:talk eta^2: .102
2/18/2019
Pittsburgh Science of Learning Center

16. Pittsburgh Science of Learning Center
Summary of results
Normal learning
Study Strategy: SE < Paraphrase
Example Type: Complete = Incomplete
Robust learning
Retention: CSE ≤ CPP
Transfer items: SE </= Paraphrase
Acc. Future Learning: SE < Paraphrase
2/18/2019
Pittsburgh Science of Learning Center

17. Pittsburgh Science of Learning Center
Outline
Introduction
Objectives
Study
Interpretation
Micro-level
Macro-level
Next
2/18/2019
Pittsburgh Science of Learning Center

18. Macro-level interpretation
Self-explanation works because
Self-explaining is a kind of personalized monologue, which is a kind of sense-making process that yields transfer.
Self-explanation is fundamentally a self-supervised learning pathway because it is generated by the individual student.
Self-explanation creates higher feature validity (‘deep’) knowledge components in the original domain that can be used to learn/construct analogous deep knowledge components in the new domain.
Simply talking wasn’t enough. It was the type of talk that changes their processing of the information. Repeating information does not necessarily link the incoming information to prior knowledge. Sense-making is about fitting new information into prior knowledge structures. The process of explaining facilitates this process whereas repetition does not.
Original Domain: electrodynamics
New Domain: magnetism
Self-
Explanation
Deep Feature
Acquisition
construct
Analogous deep
Knowledge components
2/18/2019
Pittsburgh Science of Learning Center

19. Macro-level explanation
Self-explanation provides a mechanism to construct infer-ences, resulting in better KCs, which then improves foundational skills & robust learning
Robust Learning
Outcomes: Knowledge, reasoning & learning processes
Foundational Skills
Sense-Making
Learning Processes:
Construction, elaboration, discrimination
Refinement of Features
Streng-thening
Co-Training
Instructional Processes: (independent variables or treatments)
Multiple inputs, representations, strategies …
Tutorial dialogue, peer collaboration …
self-explanation
Feedback, example variability, authenticity …
Schedules, part training …
2/18/2019
Pittsburgh Science of Learning Center

20. Which of the 8 main paths probably explain the results?
Compared to paraphrasing, self-explanation probably increased the number of correct, deep KCs
These raised scores on transfer items and the explanation parts of the normal items.
Sense- making outcomes:
More correct conceptual KCs
Deeper KCs facilitate rederivation
Deeper KCs facilitate adaptation
Deeper KCs  learning by oneself
Foundational skill outcomes:
More correct skill KCs
Stronger KCs
More general KCs
Stronger KCs leave cognitive headroom
Measurable outcomes:
Higher normal test scores
Longer retention
Further transfer
Better future learning
2/18/2019
Pittsburgh Science of Learning Center

21. Pittsburgh Science of Learning Center
Acknowledgements
Instructors
Donald J. Treacy, USNA
Robert N. Shelby, USNA
Programming
Brett van de Sande, LRDC
Anders Weinstein, LRDC
Database Assistance
Min Chi, LRDC
Michael A. Ringenberg, LRDC
Sponsors
2/18/2019
Pittsburgh Science of Learning Center