1. Deliberate practice helps students develop science process and metacognitive skills Michelle Withers Associate Professor Department of Biology, WVU

2. My history at the National Academies Summer Institutes on Undergraduate Education 200420202012 2016 2008 1 st regional SI NASI@WVU MoSI @ 23 Host Institutions in US Courtesy of J. Blum, UMN Inaugural NASI UW- Madison Joined WVU faculty

3. EnGaugements When you ask a student to do something, they are simultaneously engaged in learning and can gauge their progress by whether or how well they can perform.  Handelsman et al. Scientific Teaching.

4. Questions in my research program fall into two broad categories Pedagogical interventions in the classroom Professional development for current & future faculty

5. Acknowledgments Research group: – Undergraduates: Hayley Leight Hamza Kazmi Jennifer Winkler – Graduate students: Karen Bailey Nick Wilbur Bia Vianna Chet Saunders Collaborator – Jim Belanger Funding – NSF, WVU ARTS

6. “If they sit and listen, what they learn to do is sit and listen.”- Randall Phillis, UMass Amherst

7. Deliberate Practice The one doing is the one learning. (Ericsson et al., 1993)

8. Deliberate practice in action I.As formative assessment – Impact on graph interpretation skills II.As professional development – Impact on metacognitive skills III.Trying to make summative assessments more formative – Impact of collaborative testing on retention

9. “Ongoing assessment plays a key role – possibly the most important role – in shaping classroom standards and increasing learning gains.” Black and Wiliam, 1998 Impact of deliberate practice on graph interpretation skills by students in upper division vertebrate physiology and neurobiology courses - Primarily senior biology majors - ~ 90% identify as pre-meds

10. Senior biology majors were failing to master graph interpretation skills finalexam2exam1 100 75 50 25 0 Grade on graph interpretation exam questions (%) Data taken from a class in vertebrate physiology in which >90% of the students were biology majors in their senior year. Points represent mean ± S.E.

11. With phlogiston Without phlogiston Flexdoodle production (kg/hr) Example of pre-test graph

12. t i o n ( k g / h r ) With phlogiston Without phlogiston Types of answer% “more flexdoodles are produced in the presence of phlogiston” 85 “there’s a threshold for the effect of phlogiston”58 “the threshold = 10 0 "46 “the threshold = 1"15 “the threshold = 0"10 “the rate of flexdoodle production increases as the concentration of widgets increases” 21 “phlogistion doubles the production of flexdoodles”20 “flexdoodle production is unaffected by widget concentration below a threshold” 15 “there’s a linear relationship between flexdoodle production and widget concentration” 14 “phlogiston increases the maximum rate of flexdoodle production” 13 “the graph shows the relationship between flexdoodles and widgets” 13 “widget concentration depends on flexdoodle production” 10 “there’s an exponential(logarithmic) relationship between flexdoodles and widgets” 8 “widgets are required for flexdoodle production”5 some form of quantitative answer (other than a value for the threshold) 5

13. Pre-testExam 1Exam 2Final exam Grade on graph description questions (%) Deliberate practice through group learning exercises improve graph interpretation skills

14. “The primary goal is to help students learn to think about their own thinking so they can use the standards of the discipline or profession to recognize their shortcomings and correct their reasoning as they go.” Bain, What the Best College Teachers Do, 2004. Deliberate Practice as professional development Impact on metacognitive skills of undergraduate teaching/learning assistants for an active learning course

15. UGTAs for a course taught with scientific teaching find their experience highly valuable Item StatementMean ± SE 1. I developed a firmer grasp of the material covered in introductory biology courses. 5.83 ± 0.10 2. I better understand core biology concepts. 5.83 ± 0.10 3. I gained a better broad understanding of the field of biology. 5.74 ± 0.11 4. I improved my comprehension of material that I had not fully understood earlier in my academic career. 5.65 ± 0.12 5. I corrected misconceptions (incorrect understandings) I had previously held about biology concepts. 5.52 ± 0.14 6. I made new connections among concepts as part of the effort to understand the material more deeply. 5.70 ± 0.12 7. I thought more deeply about the material the second time around. 5.83 ± 0.10 8. I understand the discipline in a more profound way that I had before. 5.83 ± 0.10 9. I understood the material better after explaining the concepts to others. 5.70 ± 0.13 10. I applied previously learned material in new ways. 5.70 ± 0.13 11. I began to pay more attention to my own learning strategies. 5.48 ± 0.19 12. I examined my own problem-solving skills and analytical skills. 5.65 ± 0.14 13. I have performed better in subsequent biology courses as a result of the experience. 5.65 ± 0.14 Likert scale 6 = strongly agree, mean ±s.e.

16. UGTAs averaged significantly higher scores on Regulation of learning components on the MAI (MAI – metacognition awareness inventory, Schraw & Dennison, 1994) *

17. Among the Regulation components, UGTAs scored significantly higher in Planning, Management and Comprehension Monitoring * * *

18. Does collaborative testing improve content-retention in a large introductory biology course using computer-based testing? Leight, et al., 2012 CBE-LSE (11), 1-10.

19. Performance in the collaborative exam is better than the individual exam. individual group

20. Performance on cumulative questions does not differ between the individual and group testers. individual group

21. Collaborative testing doesn’t differentially affect students at different grade levels. individual group

22. Competency/Proficiency-Based Assessment 2007-2013 – 5 computer-based exams, cumulative 2014-2015 – 7 online practice exams, 2 tries each, second try is graded; 3 in-class exams 2016 – Denominator-less grading – Practice exams modularized by topic – Grades on practice an din-class tests

23. 2015 grade distribution for majors introductory biology

24. Thank you.