1. Testing in the classroom: Using tests to promote learning Richard P. Phelps Universidad Finis Terrae, Santiago, Chile January 7, 2014

2. Q. What is a standardized test? A. An assessment with at least one aspect – in its content or administration – standardized. Q. What is the key advantage of standardized testing? A. It is standardized.

3. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 3 Meta-analysis A method for summarizing a large research literature, with a single, comparable measure.

4. John Hattie’s meta-analyses of meta-analyses

5. John Hattie’s list Acceleration Classroom behavioral techniques Vocabulary programs Repeated reading programs Creativity programs Student prior achievement Self-questioning by students Study skills Problem-solving teaching Not labeling students Student-centered teaching Classroom cohesion Pre-term birth weight Peer influences Classroom management techniques Outdoor-adventure programs Home environment Socio-economic status 1. 11. 21. 31.

6. © 2012, Richard P PHELPS The effect of testing on student achievement: 1910-2010 Richard P. PHELPS

7. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 7 The effect of testing on student achievement 12-year long study analyzed close to 700 separate studies, and more than 1,600 separate effects 2,000 other studies were reviewed and found incomplete or inappropriate lacking sufficient time and money, hundreds of other studies will not be reviewed

8. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 8 Studies included in the meta-analyses 2.…when: a test is newly introduced, or newly removed quantity of testing is increased or reduced test stakes are introduced or increased, or removed or reduced

9. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 9 Number of studies of effects, by methodology type Methodology type Number of studies Number of effects Quantitative177640 Surveys and public opinion polls (US & Canada) 247813 Qualitative245 TOTAL6691698

10. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 10 Effect size: Interpretation d between 0.25 & 0.50  weak effect d between 0.50 et 0.75  medium effect d more than 0.75  strong effect

11. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 11 Which predictors matter? Treatment Group … Mean Effect Size … is made aware of performance, and control group is not+0.98 … receives targeted instruction (e.g., remediation)+0.96 … is tested with higher stakes than control group+0.87 … is tested more frequently than control group+0.85

12. Why tests? ● Students tend to study more, and learn more, when: they know they will be tested, but not precisely what will be tested »(e.g.) Experiment comparing gains of students with “take-home tests” to those with “in class tests” -- the latter learned substantially more. when there is reinforcement of material already studied ● Mastery learning experiments of 1960s—1980s: »Students learn more when asked to recall what they have learned. »Up to a point, the more students are made to actively process information, and describe it to others, the better they learn.

13. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 13 Surveys and opinion polls: Regular standardized tests, performance tests Regular tests (N ≈125) Performance tests (N ≈ 50) Respondent opiniondd Achievement is increased1.0 …weighted by size of study population1.90.5 Instruction is improved1.0 …weighted by size of study population0.9 Tests help align instruction1.0 …weighted by size of study population0.50.9

14. © 2012, Richard P PHELPSWorld Association of Education Research, 17th Congress, Reims, June, 2012 14 Qualitative studies: Effect on student achievement Direction of effect Number of studiesPercent of studies Percent without the inferred Positive2048493 Positive inferred2410 Mixed522 No change834 Negative311 TOTAL244100 244 studies conducted in the past century in over 30 countries

15. “Repeated retrieval during learning is the key to long-term retention.”

16. 10 benefits of testing and their applications to education Roediger, Putnam and Smith Direct effects of testing SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. Retrieval practice during tests enhances retention of the retrieved information (relative to not testing or even to studying) -- the “testing effect” Repeated retrieval produces knowledge that can be retrieved flexibly and transferred to other situations On open-ended assessments (e.g., essay tests) retrieval practice induced by tests helps students organize information into a coherent knowledge base. Repeated retrieval leads to easier retrieval of related information

17. 10 benefits of testing and their applications to education Roediger, Putnam and Smith Indirect effects of testing SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. Students tested frequently study more and with more regularity. Tests permit students to discover gaps in their knowledge and adjust their study efforts to focus on difficult material. Students who study after taking a test learn more than if they had not taken a test. Students who self-test or are tested more frequently in class learn more.

18. 10 benefits of testing and their applications to education Roediger, Putnam and Smith SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. Benefit 1: The Testing Effect: Retrieval Aids Later Retention Benefit 2: Testing Identifies Gaps in Knowledge Benefit 3: Testing Causes Students to Learn More from the Next Study Episode Benefit 4: Testing Produces Better Organization of Knowledge Benefit 5: Testing Improves Transfer of Knowledge to New Contexts Benefit 6: Testing can Facilitate Retrieval of Material That was not Tested Benefit 7: Testing Improves Metacognitive Monitoring Benefit 8: Testing Prevents Interference from Prior Material when Learning New Material Benefit 9: Testing Provides Feedback to Instructors Benefit 10: Frequent Testing Encourages Students to Study

19. 10 benefits of testing and their applications to education Roediger, Putnam and Smith SOURCE: Roediger, Putnam, & Smith, Ten benefits of testing and their applications to educational practice, Psychology of Learning and Motivation, 55, 2011. Benefit 1: The Testing Effect: Retrieval Aids Later Retention Benefit 2: Testing Identifies Gaps in Knowledge Benefit 3: Testing Causes Students to Learn More from the Next Study Episode Benefit 4: Testing Produces Better Organization of Knowledge Benefit 5: Testing Improves Transfer of Knowledge to New Contexts Benefit 6: Testing can Facilitate Retrieval of Material That was not Tested Benefit 7: Testing Improves Metacognitive Monitoring Benefit 8: Testing Prevents Interference from Prior Material when Learning New Material Benefit 9: Testing Provides Feedback to Instructors Benefit 10: Frequent Testing Encourages Students to Study

20. 10 benefits of testing and their applications to education Roediger, Putnam and Smith Most teachers should be testing much more frequently, …with smaller, shorter, less consequential tests. Students learns more when they test. But learn best when the tests are “spaced”. What is the optimal lapse of time between tests? The best time to test again is just before students start forgetting the information. This time lapse is shorter with discrete material, like mathematics, than with other subjects. Some studies suggest that math students should be tested at least once a week.

21. The more high-stakes decision points, the better the student performance ? SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001

22. Quality control has proportionally greater effect in poorer countries SOURCE: Phelps, Benchmarking to the best in mathematics, Evaluation Review, 2001

23. What testing skills do teachers need… …for interpreting information from large-scale tests? Basic understanding of statistics: - distributions, mean, median, skewness, kurtosis - sampling error, measurement error - type 1 / type 2 error, statistical power - sampling (size, representativeness) Protocols to help them explain tests to others: - to students - to parents - to the media

24. What testing skills do teachers need… …for developing and administering classroom tests? Practice (with each other) in writing items / prompts / rubrics : - unambiguous, relevant, un-biased Learn the optimal frequency, spacing of tests for your subject field and grade level. Understand that useful assessment can be very simple: -e.g., save the last few minutes of each class to assess by asking students to record 2-3 concepts they learned that day

25. But, you can only know what students are learning if you assess. It is easy to know what you are teaching.