Diana Laurillard and Dejan Ljubojevic London Knowledge Lab Institute of Education, London Evaluating learning designs through the formal representation of learning patterns

A Learning Design Support Environment (LDSE) IOE/LKL Brock Craft (Technical) Sarah Gelcich (Admin) Diana Laurillard (PI) Dejan Ljubojevic (RF) Javier Calzada-Prado (Intern) Oxford Liz Masterman (CoPI) Marion Manton (CoPI) Birkbeck/LKL George Magooulas (CoPI) Patricia Charlton LondonMet Tom Boyle (CoPI) LSE Steve Ryan (CoPI) Ed Whitley Roser Pujadas (Studentship) RVC Kim Whittlestone (CoPI) Stephen May Carrie Driver (Studentship)

OUTLINE Teachers are interested in learning design and the use of technology but lack the tools they need  LDSE Planning, advice, exemplars – could it also evaluate a design? Design principles in the literature Learning design patterns – to capture pedagogic principles? Representing learning theories as an evaluative framework Evaluating the pedagogy in a learning design Testing the evaluative capability of the framework Evaluating learning designs

Design principles in the literature Good feedback practice: 1. helps clarify what good performance is - goals, criteria, expected standards 2. facilitates the development of self-assessment (reflection) in learning 3. delivers high quality information to students about their learning 4. encourages teacher and peer dialogue around learning 5. encourages positive motivational beliefs and self-esteem 6. provides opportunities to close the gap between current and desired performance 7. provides information to teachers that can be used to help shape the teaching. (Nicol & MacFarlane-Dick, 2006)

Design principles in the literature 1. Encourage contacts between students and faculty 2. Develop reciprocity and cooperation among students 3. Use active learning techniques 4. Give prompt feedback 5. Emphasize time on task 6. Communicate high expectations 7. Respect diverse talents and ways of learning. (Chickering and Gamson, 1991) Where is the pedagogy that aids learning design? What active learning techniques? What kind of feedback? Can pedagogical patterns assist with designing and evaluating the quality of a learning design?

Learning design patterns The LDSE( learning design support environment) aims to evaluate the pedagogy in a learning design in terms of theory We look for pattern templates that will help to evaluate pedagogy in a design A learning design pattern as a learning activity sequence designed to lead to a specific learning outcome

Comparing pedagogy pattern templates iCOPERPlanetTELL Author & CopyrightCredits Summary/ ThumbnailContext Rationale Subject/ DisciplineContext Learning outcomesProblem Forces Group size Duration (part) Learner CharacteristicsAudience Type of Setting Graphical RepresentationDiagram Sequence of Activities Solution Roles Type of Assessment Where is the effective pedagogy captured?

We need categories within the ‘solution’ or ‘sequence of activities’ to help teacher: Develop reciprocity and cooperation among students Use active learning techniques Give prompt feedback Some accounts of learning activity sequences are quite elaborate Categories do not sufficiently discriminate the form of pedagogy that will make the critical difference to learners Comparing pedagogy pattern templates iCOPERPlanetTELL Rationale Learning outcomesProblem Sequence of Activities Solution Roles Type of Assessment Needs further detailed analysis to support learning design Pattern for understanding authentic practicePattern for helping whole class benefit from individual experiences

Representing learning patterns Structured text-based accounts, e.g. iCoper, Learning Designs Text cannot be easily interpreted by a program. Computationally defined activities can be (e.g. LAMS). Therefore can be evaluated against theory. Box-and-arrow diagrams for a sequence of activities, e.g. LAMS, iCoper, ISiS-ECTEL

Representing learning theories The Conversational Framework: developed to provide a representation of what it takes to learn in education based on the main design principles for learning and pedagogy drawn from the literature can test a learning design in terms of which aspects of the framework it covers.

The learner learning LC TC LP TP Thoughts Action plans Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Reflection Producing Revising Working to a goal Feedback LC LP Adaptation being supported,actingthinking,

Learning theories represented in the Conversational Framework LC TC LP TP Thoughts Action plans Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Reflection Producing Revising Working to a goal Feedback LC LP Adaptation 1. Encourage contacts between students and faculty 4. Give prompt feedback 2. Develop reciprocity and cooperation among students 3. Use active learning techniques

Learning theories represented in the Conversational Framework LC TC LP TP Thoughts Action plans Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Acting Listening/ Reading Reflection Producing Revising Working to a goal Information LC LP Adaptation Reflection Adaptation

Learning theories represented in the Conversational Framework LC TC LP TP Thoughts Action plans Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Acting Listening/ Reading Reflection Producing Revising Working to a goal Information LC LP Adaptation Reflection Adaptation

Learning theories represented in the Conversational Framework LC TC LP TP Thoughts Action plans Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Acting Listening/ Reading Reflection Producing Revising Working to a goal Feedback LC LP Adaptation Reflection Adaptation

Representing technologies in support of learning LC TC LP TP Thoughts Action plans OC OP Reflection LC LP Adaptation

Evaluating a learning design The Conversational Framework: can test a learning design in terms of which aspects of the framework it covers - IF we can link each LAMS activity (or similar computationally defined activity) in the sequence to part of the framework

LAMS – representing learning design as a temporal sequence of computationally defined activities Chat Chat and Scribe Forum Forum and Scribe MCQ Notebook Noticeboard Q&A Resources & Forum Share resources Submit Voting To understand the processes within a system

Chat and Scribe Mapping LAMS activities to theory via CF LC TC LP TP Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Producing Revising Working to a goal Feedback LC LP Forum and Scribe MCQ Notebook Q&A Share resources Submit Voting Chat and Scribe Notebook Noticeboar d Q&A Share resources Submit Forum and Scribe MCQ Voting

LAMS – representing learning design as a temporal sequence of computationally defined activities To understand the processes within a system

Interpreting the quality of the design LC TC LP TP Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Producing Revising Working to a goal Feedback LC LP Notebook Chat and Scribe Noticeboar d Q&A Share resources Submit Forum and Scribe MCQ Voting System can interpret design as scoring: 4 out of 10 activities; No iterations; Time for each task is good; Sequence is good for ‘awareness’; less good for ‘understanding’ Evaluating the quality of the design

Improving the quality of the design LC TC LP TP Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Producing Revising Working to a goal Feedback LC LP Chat and Scribe Noticeboar d Q&A Voting System can propose: Adding in designs or patterns that complete the other cycles; Checking that the sequence follows the motivating cycles of iteration

Testing the framework The Conversational Framework: can differentiate between sequences in terms of coverage of the framework, amount of iteration, and effective ordering of activities But to what extent can it recognise the relative effectiveness of learning designs identified in the literature?

Pattern 1 Contrasting cases text Analyse data S analyses data S produces graphs S listens to lecture S selects and predicts Very accurate MCQs High transfer to prediction Pattern 2 Theory and examples text Read text S reads text S produces summary S listens to lecture S selects and predicts Very accurate MCQs Much lower transfer to prediction Comparing learning design patterns Intended learning outcome: To predict accurately the outcomes for a hypothetical experiment on memory [Schwartz and Bransford,1998] Learning activities Reading T sets a task goal Investigating Producing output Listening Working to achieve goal Actual outcomes Pattern 1 Contrasting cases text Analyse data S analyses data S produces graphs S listens to lecture S selects and predicts Very accurate MCQs High transfer to prediction Pattern 2 Theory and examples text Read text S reads text S produces summary S listens to lecture S selects and predicts Very accurate MCQs Much lower transfer to prediction Critical differences between patterns are not just the nature of the activity, but the internal relations between contents of activities in the sequence

Interpreting the quality of the designs LC TC LP Text Data Guidance 3. Summarising 3. Investigating 1. Reading 4. Producing Revising 2. Working to a goal Feedback LC LP System must be able to differentiate between producing graphs and summaries; And between providing Text and Data in the practice environment 5. Listening 4. Producing Reflecting Adapting Asking Questions

Representing the range of good pedagogy LC TC LP TP Guidance OC OP Articulating ideas Others’ ideas Preparing Outputs Others’ Outputs Asking Questions Investigating Listening/ Reading Producing Revising Working to a goal Feedback LC LP Notebook Chat and Scribe Noticeboar d Q&A Share resources Submit Forum and Scribe MCQ Voting A System like LAMS can represent good pedagogy in terms of type and range of learning activity Cannot represent good pedagogy within the practice environment – simulation, model, digital library, data repositories, type of task and type of feedback

Summary Design principles in the literature Learning design patterns Representing learning theories as a framework for design Evaluating the pedagogy in a learning design Testing the evaluative capability of the framework Too general to be helpful for teacher-designers May miss categorising the critical pedagogy Makes it possible to relate theory to computational activities Needs mapping from framework to LAMS or similar system May not capture all the key differences in effectiveness

Pedagogy operates both across and within activities We need user-editable patterns for activity sequences – One role for learning technologists is to create the means for teachers to adopt and adapt pedagogic patterns for activity sequences – e.g. LAMS – One role for teachers is to these to adopt, adapt and share pedagogic patterns for activity sequences We need customisable programmed patterns for activities – Another role for the learning technologist is to discover and create the means for teachers to adopt and adapt pedagogic patterns for practice environments – e.g. GLO tool, NetLogo Evaluating learning designs through the formal representation of learning patterns Made publicly available by ALT under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales license