Techniques for Efficient Learning Jack Leggett

Catch - Free Film Sheet Press Clothes Heavy Music Judge Bench

Learning interventions should be… Concrete Cost-effective Supported by high-quality evidence Concrete: Consist of specific instructions, not only abstract principles (e.g., “use active learning” with no instruction on how to do that or clear definition of “active”) Cost-effective: For a given investment of resources (money, work-time by teachers or students), the technique should produce a measurably better outcome than other things that could be done with the same resources Evidence: Cost-effectiveness should be supported by a weight of evidence from multiple realistic studies, including at least some experiments (see next slide)

Experimental Treatment Alternative Treatment No Special Treatment Recruit participants Randomly assign to... Experimental Treatment Alternative Treatment No Special Treatment Design of one kind of experiment (a classic randomised controlled trial) Crucial features include random assignment (so that differences between conditions can be attributed to the treatments, not to the pre-existing traits of participants), the use of a placebo control group (“alternative treatment”) to account for placebo effects or other reactivity, and measurement of long-term outcomes (as many short-term advantages vanish in the long term, especially learning effects) Another important thing to consider is the realism of the experimental situation – does it use realistic participants, learning material, teachers, and classrooms? Anyone suggesting major changes to teaching practices should be asked to support their suggestion with evidence from studies of this kind. A qualified expert will be able to provide it, or explain the best available evidence from other kinds of study. Measure and compare outcomes of interest in the long-term

*** - Asked audience to attempt to recall word-pairs

Catch - Free Film Sheet Press Clothes Heavy Music Judge Bench - Ask audience to count number they got correct

Forgetting curve (Ebbinghaus, 1913)

- Revision refreshes memory and slows forgetting

*** Asked audience members to draw both faces of dollar coin from memory

Note positioning and content of text, facing of queen, animal shown, number of animals, etc.

*** - Asked audience to write down coin colours and sizes, and note colours

Note that memory for sizes, colours, and which denominations exist was better than for visual details on faces of coins/notes Remembering visual details involves more complex information, but also information that is not needed for actually using the coins and notes

People do not reliably learn information from mere exposure But they do reliably learn information needed for tasks they perform Strong learning results not from mere exposure, but from overcoming increasingly difficult, realistic challenges and receiving clear feedback Note in coin case you get clear feedback either from other info on coin or from result of transaction In subject like maths the tasks are obvious, but what about learning factual information? What kind of increasingly difficult challenges can learners overcome there?

People do not reliably learn information from mere exposure But they do reliably learn information needed for tasks they perform Strong learning results not from mere exposure, but from overcoming increasingly difficult, realistic challenges and receiving clear feedback Note in coin case you get clear feedback either from other info on coin or from result of transaction In subject like maths the tasks are obvious, but what about learning factual information? What kind of increasingly difficult challenges can learners overcome there?

A learning task for developing knowledge and understanding Retrieval Practice A learning task for developing knowledge and understanding - Any deliberate retrieval of information from memory is retrieval practice. Examples include all the memory tasks in this talk (recall word-pairs, recall coin faces, sizes, colours, and note colours)

Roediger & Karpicke (2006) Participants read passage of text Then reread or take practice test Then freely recall facts from passage Participants were university students Material was textbook-like, factual material No feedback was given on the practice test, so in the rereading situation participants were re-exposed to all information, but in practice test (retrieval practice) situation they were re-exposed only to the info they could recall.

Roediger & Karpicke (2006) Notice normal forgetting pattern after reading revision, but slowed forgetting after retrieval practice Many studies find that reading is superior to retrieval practice when the information only needs to be remembered for a short time, and this is probably part of the reason that students often believe reading is more effective for revision. In general people seem more sensitive to the short-term effects of learning techniques rather than the long-term effects.

Forms of retrieval practice

Recall all word-pairs in any order Free Recall Recall all word-pairs in any order - Essay questions in exams require free recall, as students are given a single prompt and required to recall a large amount of information.

Recall the other word in the pair Cued Recall Recall the other word in the pair Heavy - ? - Short-response questions often require cued recall (recalling a complementary piece of information given a cue)

Which of these pairs was in the list? Recognition Which of these pairs was in the list? Judge - Bar Pressure Crush Film Sheet Free Cost - Multiple-choice tests use recognition memory in part, though well-designed MCTs often require recall as well, as they include familiar but incorrect alternatives. In this example, many of the incorrect alternatives are similar to items that were in the list (e.g., Judge-Bar instead of the correct Judge-Bench).

Adjusting challenge Form of retrieval task Free recall (create from memory an essay, paragraph, concept map, etc.) Cued recall (short answer question) Recognition (multiple choice question) Hints and follow-up questions (“scaffolding”) Free recall tasks are usually harder than cued recall tasks, which in turn are usually harder than recognition tasks. “Harder” in this case meaning that retrieval of a given piece of information is less likely to be successful or is slower. Experiments show that more difficult retrieval tasks, if performed successfully, are more effective for learning than easier retrieval tasks. This aligns with the challenge principle mentioned earlier.

Adjusting challenge Hints Heavy - M _ _ _ c - Note that multiple hints can be given to make retrieval easier. In an extreme case, hints could be added until the retrieval task was trivially easy. Note though that experiments generally find that even a trivially easy retrieval task, though less effective than a challenging one, is still more effective than simply reading (or otherwise passively reviewing) the information. Hint: Something you can listen to.

Adjusting challenge Form of retrieval task Free recall (create from memory an essay, paragraph, concept map, etc.) Cued recall (short answer question) Recognition (multiple choice question) Hints and follow-up questions (“scaffolding”)

Adjusting challenge Form of retrieval task Free recall (create from memory an essay, paragraph, concept map, etc.) Cued recall (short answer question) Recognition (multiple choice question) Hints and follow-up questions (“scaffolding”) Spacing Time between repetitions of any learning task - Increasing spacing increases task difficulty, but also increases learning from successful performance of that task. This runs against an intuition many people hold that performing many repetitions in a short time is good for learning – in fact, unless performance is improving over those repetitions, massed practice of this kind often has almost no effect on long-term learning. For example, reading the same information five times in quick succession is almost no better than just reading it once or twice, but reading it once each day for five days is very effective.

Adjusting challenge Spacing Learning material divided into five sets On day 1: Studied set A four times and set E once On day 2: Studied set B four times and set E once On day 3: Studied set C four times and set E once On day 4: Studied set D four times and set E once On day 5: Took test on all material How would performance differ between the sets? Notice that all sets were reviewed four times. Only the distribution of the repetitions, and the delay between the review sessions and the test varied between sets. Set D was reviewed four times the day before the test, and set E was reviewed once the day before the test and three additional times earlier. If massing repetitions is best for learning, performance should be better with set D than set E. Kornell (2009)

Adjusting challenge Spacing Massed (4 times on one day) Spaced (once each day) Spacing is a helpful challenge in learning - Notice the “cramming” effect for set D. Cramming works, but, as can be seen by performance with sets A-C, its benefits are short-lived. Spacing has both a greater effect on learning, and its effect lasts longer. Kornell (2009)

Interleaving, a friend of spacing Blocked Randomly interleaved A A A B B B A B A B B A B A B A B A … - Here, A and B stand for different learning tasks. For example, A and B could be maths problems requiring different formulae. Or, each A could represent one revision of a given piece of information, and each B a revision of a different piece. Finally, each A could represent a different example from the same category (e.g., different paintings by the same artist, different birds of the same species, different bowls of the same type in cricket) and each B could represent a different example of another category. Note that the crucial thing is that repeated exposures to one piece of information (type of maths problem, painting style, fact to learn) are either presented in a block, or interleaved with exposures to another kind of information (different type of maths problem, painting style, fact). Interleaving here does NOT refer to mixing different domains of learning (i.e., doing some maths, then some history, then more maths, then more history), it refers to how repetitions of the same piece of information are arranged with respect to repetitions of another piece of information.

Interleaving, a friend of spacing Interleaving imposes spacing, encourages retrieval, and improves learning of categories To increase challenge, change from a blocked schedule to an interleaved schedule - Note that the crucial thing is to make the level of challenge appropriate for the learner at their current level of ability. If learners need blocked practice at first in order to have some success in the task, they should be given blocked practice. But, as their ability improves, interleaving can be used to increase challenge. As with all the ways of adjusting challenge, there is no need to start at the hardest setting – the idea is just to move to harder settings (i.e., recognition to free recall, short spaces to longer spaces, blocking to interleaving) as learning progresses so that challenge is maintained and ability continues to improve.

Adjusting challenge Form of retrieval task Free recall (create from memory an essay, paragraph, concept map, etc.) Cued recall (short answer question) Recognition (multiple choice question) Hints and follow-up questions (“scaffolding”) Spacing Time between repetitions of any learning task Interleaving different learning tasks is a good way to space repetitions Scaffolding, spacing, and interleaving work for any learning task Scaffolding, spacing, and interleaving work for retrieval tasks, but also for skills (e.g., problem-solving techniques in maths or other subjects, and motor skills). The ideal rate of success is not yet known, and probably differs between tasks and between learners (some learners may find failure more demoralising than others, for example). Probably learners should have at least partial success on most of their attempts, and full success sometimes. For example, leaners should probably be producing at least some correct information on most retrieval practice questions, take at least some correct steps on most maths problems, etc. Total failure is probably not very productive, except as a way of finding out the learner’s current limits, but partial failure can improve learning compared to making no attempt (see feedback slides later). Adjust these so that learners experience challenge, but mostly succeed

Adjusting challenge Form of retrieval task Free recall (create from memory an essay, paragraph, concept map, etc.) Cued recall (short answer question) Recognition (multiple choice question) Hints and follow-up questions (“scaffolding”) Spacing Time between repetitions of any learning task Interleaving different learning tasks is a good way to space repetitions Scaffolding, spacing, and interleaving work for any learning task Scaffolding, spacing, and interleaving work for retrieval tasks, but also for skills (e.g., problem-solving techniques in maths or other subjects, and motor skills). The ideal rate of success is not yet known, and probably differs between tasks and between learners (some learners may find failure more demoralising than others, for example). Probably learners should have at least partial success on most of their attempts, and full success sometimes. For example, leaners should probably be producing at least some correct information on most retrieval practice questions, take at least some correct steps on most maths problems, etc. Total failure is probably not very productive, except as a way of finding out the learner’s current limits, but partial failure can improve learning compared to making no attempt (see feedback slides later). Teachers might consider the direct effects of success rates on learning, as well as the indirect effects via motivation and measurement of the learner’s ability. Adjust these so that learners experience challenge, but mostly succeed

Feedback

Kornell et al. (2009) Participants learn some word-pairs by reading and some by guessing with feedback 24 hours later, given cued-recall test on all pairs

Read Trial 13 seconds to read Press Clothes

Guess Trial 8 seconds to guess Catch ?

Guess Trial 8 seconds to guess Catch ? Ball

Guess Trial 5 seconds to view feedback Catch Free Note that total time was matched at 13 seconds (in guess condition 8 seconds to guess and 5 seconds to read target) Correct guessing was very unlikely – about 95% of guesses were incorrect. This was intentional because the researchers wanted to study effects of corrective feedback.

Test Trial Recall word shown by computer Catch ? - Here learners had to type in the word shown by the computer earlier

Kornell et al. (2009) Results of delayed cued-recall test Errors are an inevitable result of challenge, but usually improve learning if feedback is clear - Note that this benefit seems to depend on the guesses being somewhat “educated” (see Knight et al., 2012, Testing unsuccessfully: A specification of the underlying mechanisms supporting its influence on retention). If learners have no basis for making a plausible guess, their failures may impair learning rather than helping it. Generally learners who are not completely new to the material will be able to make some kind of informed guess though, so errors in most classroom situations are probably not harmful as long as clear feedback is given.

Challenging feedback Scaffolding Scaffolded feedback encourages learners to self-correct, and improves long-term retention Scaffolded feedback involved allowing the learner to attempt to answer multiple times during retrieval practice, and giving an extra letter of the answer after each error. Standard feedback involved simply presenting the correct answer after the first error during retrieval practice. In the classroom, scaffolded feedback can be used one-on-one easily. For whole-class activities, students can be asked to work individually at first and be given the question or task in its most difficult form – then, as they continue working some follow-up questions or hints can be given to make it easier. This allows all students to attempt the task in its most challenging form, while providing support for students who need it. Alternatively, the difficult form of the task could be set for the whole class, and scaffolding provided individually by the teacher as needed, or students can assist one another working in groups. Finn & Metcalfe (2010)

Challenging feedback Explanations Explanatory feedback improves transfer of learning to new situations (i.e., understanding) Here “Repeated” questions appeared in both the practice test (retrieval practice session) and the final test. “New Inference” questions appeared only in the final test, but relied on information revised during the practice test. Butler et al. (2012)

Summary Whenever teaching, consider how to use a task rather than mere exposure Retrieval a good task for developing knowledge Use hints and spacing to encourage success with challenge, not fluency Give scaffolding and explanatory feedback Embrace errors as signs of challenge and chances for learning Note that many mere exposure activities can be turned into tasks. For example, instead of just presenting a list of formatting instructions for an assignment, the teacher could present the list and then have the students actually format a document of filler text. When presenting new facts or concepts, the teacher could start by asking students to give their intuitive ideas about the topic before the lesson, and end the lesson with a retrieval activity. When introducing a new concept, students can be encouraged to guess the next step of an explanation, or to generate examples or analogies of the concept. Or, after the explanation, they can practice classifying short vignettes as examples or non-examples of the concept and receive feedback on their attempts. Remember that spacing, scaffolding, and explanations work for all learning tasks, not just retrieval practice.

Contact: jack.leggett@uqconnect.edu.au Further reading: Bjork et al. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417-444. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58. Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., and Metcalfe, J. (2007) Organizing Instruction and Study to Improve Student Learning (NCER 2007-2004). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://ies.ed.gov/ncee/wwc/PracticeGuide/1 - Feel free to contact me if you would like to read any of the studies I described in the talk.