How the PSLC Members will Develop a Theory of Robust Learning

How the PSLC Members will Develop a Theory of Robust Learning
Primary strategy: Summarize site visit presentation (which they don't have), informed by site visit response, and use quotes from site visit report We said in response we would: 1) identify fast track courses -- what are they? - algebra & geometry are farthest along 2) id fast track technology - Cog tutor authoring tools? PSLC Directors: Ken Koedinger and Kurt VanLehn 9/16/2018 Pittsburgh Science of Learning Center

Pittsburgh Science of Learning Center
PSLC Vision Problem: Not enough rigorous, real-world, generalizable results Solution: LearnLab Technology & social process resources New paradigm for learning experimentation & theory Intellectual merit New understanding of robust learning Test of lab-based learning principles in real settings Use-inspired hypotheses to test in the lab Broad impact Wide use of new experimental paradigm More effective courses New evidence-based education Here is the main argument for the center. Despite some promising efforts, like those reported in the National Research Council’s “How People Learn” report, we do not have enough rigorous real-world, generalizable results in the Learning Sciences. We propose “LearnLab”, a resource for learning researchers that combines technology & social process resources to facilitate a new paradigm for learning science experimentation and theory development. The LearnLab will support researchers in our center and worldwide in developing a new understanding of … The center will produce broad impact through … 9/16/2018 Pittsburgh Science of Learning Center

LearnLab Addresses Current Limitations of Learning Experiments
Rigorous studies with real content in real classrooms with real students Collect & analyze fine-grain data over long durations Measure robust learning Long-term retention, transfer, accelerated future learning 9/16/2018 Pittsburgh Science of Learning Center

The 5 R’s of LearnLab More Rigorous and Realistic Research Results on Robust Learning! 9/16/2018 Pittsburgh Science of Learning Center

Outline for today’s talk
Motivation: Why a theory? Meta: What kind of theory? Initial theory Micro Macro Future work 9/16/2018 Pittsburgh Science of Learning Center

Interacting Pieces of PSLC
Principles of Robust Learning Studies Learning Theory LearnLab Courses Enabling Technology 9/16/2018 Pittsburgh Science of Learning Center

SLC-wide motivations Questionable accretion Compare textbooks’ contents Why no problem sets in textbooks? Compare to other applied sciences Medicine Agriculture 9/16/2018 Pittsburgh Science of Learning Center

Motivation: Why a theory? Meta: What kind of theory? Initial theory Micro Macro Future work Next 9/16/2018 Pittsburgh Science of Learning Center

Feasible types of theory
Newell’s A single, computational model An accurate simulation of a student Simon’s Concepts e.g., working memory Combine them to form explanations 9/16/2018 Pittsburgh Science of Learning Center

Key Conceptual Premises & Vocabulary
Learning is partially decomposable Knowledge component: a piece of acquired knowledge Concept, principle, production rule, schema, reasoning process, meta-cognition, ... Knowledge event: an interval in time or a piece of instructional material where a knowledge component might be learned/used Knowledge component learning Construction: Component is (re-)derived from perceptions and existing knowledge Feature validity: Response-relevant or deep features acquired, irrelevant/shallow features ignored Strength: Accumulated importance, ease of retrieval, efficiency of processing 9/16/2018 Pittsburgh Science of Learning Center

Illustrations of knowledge components
False: “massless” is missing The magnitudes of the tension forces exerted by the ends of a taut string are equal. A Chinese character, its pronunciation and its meaning The number of kisses exchanged when French people greet is meaningful If I’m feeling a bit uncertain, attempt the step; but if I’m feeling quite uncertain, ask for hint. Skills Overly general Meta 9/16/2018 Pittsburgh Science of Learning Center

Illustrations of knowledge events
Read definition Study example Use in proof Explain to another student Make error, get tutored, correct Start to write def, look up in text, finish 9/16/2018 Pittsburgh Science of Learning Center

Basic knowledge events
Description: Verbal rule, explanation provided Self-explanation: Verbal rule, explanation generated Example: Instance of stimulus-response, situation-action, features-label provided Practice: Stimulus/situation/ features provided, student generates response/action/ label Passive Active Explicit Description Self-explanation Implicit Example Practice 9/16/2018 Pittsburgh Science of Learning Center

How Research Clusters Produce Learning Outcomes
Instructional treatments vary knowledge events to produce changes in construction, feature validity, or strength of knowledge components Feature Validity Construction Strength Knowledge Events: Descriptions, Examples, Self-Explanation, Practice Refinement of Features Dialogue Co-Training Fluency 9/16/2018 Pittsburgh Science of Learning Center

Knowledge events over time
While studying example, tries to self-explain; fails; looks in text; succeeds While solving problem, looks up example; recalls explanation; maps it to problem Recalls explanation; slips; corrects Knowledge event duration Solves without slips Solves without slips First Second Third Fourth Fifth 9/16/2018 Pittsburgh Science of Learning Center

Illustrations of “Construction”
Natural language understanding: “A surface always exerts a force that is perpendicular (“normal”, in mathematics) to the surface. It is called the normal force on the object.” Analogy “When an object presses on spring, the spring pushes back. A surface is like a very stiff spring. When on object pushes on it, it pushes back. This is called the normal force on the object.” Induction: “Here are some normal forces:” 9/16/2018 Pittsburgh Science of Learning Center

Illustration of “feature validity”
At rest; 5 kg block; 1 kg block; Frictionless What is big block’s velocity after falling 2 m? Surface features match training problems for Newton’s second law Pulley, string, blocks Deep features match training problems for Conservation of Energy Two times, no friction forces, no applied forces, displacement, velocity 9/16/2018 Pittsburgh Science of Learning Center

Micro learning processes & results
Construction Existance Refinement of features Feature validity Strengthening Strength 9/16/2018 Pittsburgh Science of Learning Center

Why is a macro level necessary?
Micro level describes learning processes, but not why they occur. What kinds of situations, instruction, backgrounds, etc. tend to elicit micro learning, and hence robust learning? Need a taxonomy… 9/16/2018 Pittsburgh Science of Learning Center

Original Research Clusters & Pathways to Robust Learning
Foundational Skills Sense-Making Co-Training, Multiples Refinement of Features Dialogue Fluency 9/16/2018 Pittsburgh Science of Learning Center

Differentiating Processes & Outcomes
Instructional Process Learning Process Outcome: Knowledge/ reasoning process Robust Learning Sense-Making Foundational Skills Co-Training, Multiples Refinement of Features Dialogue Fluency

Elaborating Missing Processes
Robust Learning Outcomes: Knowledge, reasoning & learning processes Foundational Skills Construction, elaboration, discrimination Co-Training Refinement of Features Streng-thening Learning Processes: Sense-Making Instructional Processes: Multiple inputs, representations, strategies Tutorial dialogue, peer collaboration Feedback, example variability, authenticity Schedules, part training

Instructional, learning, & reasoning processes
Instructional Processes Reasoning Processes Outcomes: Reasoning & Learning Processes Learning Processes Learning Processes Test of accelerated future learning Pre-test Post-test of near & far transfer Long-term retention test Robust Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Illustrations from PSLC Studies
Katz Liu & Perfetti Pavlik Instruction Learning Process Outcome 9/16/2018 Pittsburgh Science of Learning Center

Katz: Post-practice reflection
Instruction: After solving a physics problem, student answers reflection questions, e.g., What is the basic approach? What if the block were moving up instead of down? Learning processes: Construction, e.g., basic approach is Conservation Refinement, e.g., initial block direction is irrelevant Outcomes: Existance of “basic approach” knowledge components Feature validity of many knowledge components Manipulation: Dialogue vs. text remediation Dialogue increases frequency of construction & refinement 9/16/2018 Pittsburgh Science of Learning Center

Liu & Perfetti: Chinese character learning
Task: Correctly pronounce Chinese characters Instructional treatment: Multiple inputs Show video of mouth movements in addition to just audio of sound of the character [Other possible inputs & outputs: still of lips, pin yeng (written version of pronunciation), meaning] Learning process: Co-training Feature refinement Different media make different features salient Outcome: Knowledge components with better feature validity 9/16/2018 Pittsburgh Science of Learning Center

Pavlik: Scheduling practice
Task: given French word, provide English word Instruction: Vary training schedule Use ACT-R to compute sequence that optimizes costs & benefits of training task difficulty Spacing between practice trials increases as student gets better at word pair Learning process: Strengthening through practice Outcome: Knowledge components with higher strength 9/16/2018 Pittsburgh Science of Learning Center

Pathways lead to measures
Robust learning pathways yield measurable outcomes through well-acquired knowledge components (including both domain & meta knowledge components) Pathways: Sense-making Foundational Skills Long-term retention Transfer Future Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Robust learning pathways yield measurable outcomes through well-acquired knowledge components (including both domain & meta knowledge components) Pathways: Sense-making Rederivation Foundational Skills Strengthening Long-term retention Transfer Future Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Robust learning pathways yield measurable outcomes through well-acquired knowledge components (including both domain & meta knowledge components) Pathways: Sense-making Rederivation Adaptation Foundational Skills Strengthening Deep feature perception Long-term retention Transfer Future Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Robust learning pathways yield measurable outcomes through well-acquired knowledge components (including both domain & meta knowledge components) Pathways: Sense-making Rederivation Adaptation Self-supervised learning Foundational Skills Strengthening Deep feature perception Cognitive headroom Long-term retention Transfer Future Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Encourage Active Declarative Processing Through Self-Explanation
Aleven, V. & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2) 9/16/2018 Pittsburgh Science of Learning Center

Problem: Shallow knowledge = “low feature validity”
Example of a knowledge component with low feature validity “Looks-equal” production rule If the goal is to find angle A and it looks equal to angle B and angle B is D degrees Then conclude that angle A is D degrees 9/16/2018 Pittsburgh Science of Learning Center

Example of Shallow Reasoning
9/16/2018 Pittsburgh Science of Learning Center

Explanation Condition
Problem solving answers Explanation by reference 9/16/2018 Pittsburgh Science of Learning Center

Problem Solving Condition

Assessing transfer: “Not Enough Info” item

Assessing transfer: Incorrect over-generalization

SE Study 2 Results 9/16/2018 Pittsburgh Science of Learning Center

Mapping to PSLC terms What robust learning measures? What robust learning pathways? Sense-making or foundational skills What knowledge components changed from pre to post? What learning & reasoning processes lead to measures? 9/16/2018 Pittsburgh Science of Learning Center

Mapping to PSLC terms Robust learning measure:Transfer What robust learning pathway(s)? Sense-making What knowledge components changed from pre to post? Higher feature validity on geometric reasoning components (More accessible of declarative knowledge?) What learning & reasoning processes lead to measures? Deep feature perception & adaptation? 9/16/2018 Pittsburgh Science of Learning Center

Instructional, learning, & reasoning processes
Instructional Processes Reasoning Processes Outcomes: Reasoning & Learning Processes Learning Processes Learning Processes Test of accelerated future learning Pre-test Post-test of near & far transfer Long-term retention test Robust Learning Measures: 9/16/2018 Pittsburgh Science of Learning Center

Tutoring Help-Seeking
Goal: Foster long-term learner independence Model of ideal learning & help-seeking behaviors Tutor this model Improve robust learning Long-term retention Transfer Accelerated future learning 9/16/2018 Pittsburgh Science of Learning Center

Measuring Future Learning

Theory development via
Discussions at research cluster meetings Reporting research Web pages Advisory board visit (Dec. 13 & 14, 2005) Annual progress report (winter, 2006) Site visit (spring, 2006) Planning Strategic plan (summer, 2006) Suggest calls for research projects Individuals & clusters engage in all of the above 9/16/2018 Pittsburgh Science of Learning Center

Evaluation of the PSLC theory development effort
Members understand the terms/theory may or may not ascribe to it C-STPS interviews PSLC documents become more integrated C-STPS text analysis Member’s papers in open literature use PSLC terms/theory Others use PSLC terms C-STPS interviews and text analysis 9/16/2018 Pittsburgh Science of Learning Center

END These are some of the issues we perceive as challenges and deserving of particular focus. We are looking forward to hearing what you see as challenges. 9/16/2018 Pittsburgh Science of Learning Center

Limitations/missing pieces
Other key learning-related processes Utility, attention, motivation, meta-cognition, perceptual chunking, declarative-procedural distinction … Computational, cognitive, social/motiv considerations Science has formulas, where are ours? generalization gradient, power law of practice, spacing, transfer gradient, cognitive IRT Machine learning theorems … The Psych of HCI type book “Model Human Learner” instead of “Model Human Processor” (see photo of white board EC meeting notes) 9/16/2018 Pittsburgh Science of Learning Center

Illustration: Self-explanation
Given examples to study then problems to solve, self-explaining (SE) the examples improves problem solving (PS) immediately, after delay, on far-transfer problems. Probably increases learning on new tasks as well. Sense-making Rederiving principles during SE encourages rederiving during PS Adapting principles during SE to apply encourages similar adapatation during PS Self-supervision: Deciding to SE and finding that it makes PS easy (easier?) encourages SE later Foundational Skills Using principles to SE strengthens them SE adds deep features to the examples Cognitive headroom: Easy recall of parts facilitates SE of whole Measures Retention Transfer Acceleration 9/16/2018 Pittsburgh Science of Learning Center

Instruction Learning Process Outcome PSLC Study Examples Katz: Post-reflection dialog in physics leads to constructions that increase knowledge component existance, and to feature refinements that increase feature validity Liu & Perfetti: Audio & lip video of chinese leads to co-training that increases feature validity Aleven: Contiguous diagram elements & labels leads to better refinement & increase feature validity Pavlik: Expanded spacing of practice of French vocabulary optimizes strengthening to strengthen 9/16/2018 Pittsburgh Science of Learning Center

Aleven: Contiguous labels enhance geometry tutor
Instruction: Givens & answers in either… a table beneath the diagram, or boxes in the diagram Learning process: Refinement Placing data near visual referents increases frequency of refinement Outcome: Feature validity 9/16/2018 Pittsburgh Science of Learning Center

Some slides from 05 site visit

Pre-PSLC Pittsburgh Theory
A mess of concepts Self-explanation Procedure vs. declarative memory Help-seeking behavior Competition (of 1st language with 2nd) Co-training … Can generate several possible explanations/predictions, but not evaluate their probability As pointed out by our Advisory Board, in the eyes of the rest of the world, Pittsburgh already has a theory. As a group, we tend to heavily use some concepts, such as the ones listed here, and avoid others, such as “cognitive load” or “negotiation.” However, the Pittsburgh theory is messy. We use terms inconsistently and redundantly. Individuals often focus on only some of the concepts, missing explanations that would seem obvious to other Pittsburghers. More concretely, although this messy theory allows generating explanations, it seldom allows us to evaluate the probability of the predictions. Herb Simon would be quite happy with this state of affairs. In his opinion, this is as good as theory gets. Allen Newell would not be so happy. He sought a much more tightly integrated theory. 9/16/2018 Pittsburgh Science of Learning Center

How to improve the theory
Talking together about our work Increase shared beliefs/concepts Build on each other’s results Grounding our terminology Theory-laden tools, e.g., Data Shop Theory-laden tutoring systems Computational models, e.g., Simulated students While recognizing that Herb Simon is right in his view that theory is a collection of beliefs and concepts used daily by a group of scientists, we believe this community-wide collection can be made more coherent and powerful by having people talk to each other more at events like this one, and by working daily with tools and models that ground some of the theoretical terms. This would make Allen Newell a little happier, although it falls short of the unified theory that he envisioned. 9/16/2018 Pittsburgh Science of Learning Center

How concrete and general will our theory become?
Models of memory, attention, … ACTR, Epic, etc. Models of skill acquisition Cascade, HS, Iccarus, etc. SimStudent project? Strategic plan’s framework Includes social/motivational Newell’s UTC book ACTR, Epic, CAP, etc. are models of low-level cognition. Cascade, HS, Icharus, etc. are modesl of high level cognition. The SimStudent project is on track to offer another such high level cognitive model with the practical application of helping authors create cognitive tutors. These models were concrete but not general. On the other hand, the Strategic plan has a section on theory that articulates a general framework that explicitly recognizes the impact of social and motivational factors on learning. Newell’s book take an even broader view. These are general but not concrete. It is conceivable that all this work could be integrated to form a computational model that is both concrete and general. How far we go remains to be seen. 9/16/2018 Pittsburgh Science of Learning Center

Multi-directional payoffs
Cognitive psychology Neuro-cognitive studies Computational models of cognition Even without a concrete, general theory, we can reasonably expect multi-directional payoffs. Intelligent tutoring systems Social & meta-cognitive studies Machine learning 9/16/2018 Pittsburgh Science of Learning Center

How the PSLC Members will Develop a Theory of Robust Learning

Similar presentations

Presentation on theme: "How the PSLC Members will Develop a Theory of Robust Learning"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

How the PSLC Members will Develop a Theory of Robust Learning

Similar presentations

Presentation on theme: "How the PSLC Members will Develop a Theory of Robust Learning"— Presentation transcript:

Similar presentations

About project

Feedback