Promoting Learning Gain and Confidence Gain in the Classroom Through Self-Assessment and Peer-instruction Dr Fabio R. Aricò @FabioArico ARU – Jun 2018

YOUR PRESENTER Fabio Aricò National Teaching Fellow Senior Lecturer in Macroeconomics School of Economics – University of East Anglia, UK Research fields Higher Education policy and practice (widen. access, satisfaction) Technology Enhanced Learning Self-Assessment and Academic Self-Efficacy Twitter: @FabioArico

ACKNOWLEDGEMENTS HEA – Teaching Development Grant Scheme HEFCE Piloting and Evaluating Measures of Learning Gain UEA Students, Alumni, and Research Assistants

AIMS & OBJECTIVES My talk will be all about pedagogies which I have designed, used, and evaluated. They are now being rolled across different disciplines. Sharing my practice  take home at least one idea! Inspiring research  from research-led teaching to teaching-led research Support developments in the HE sector  in the UK: TEF is coming (learning gain) Support personal development  promoting and evidencing excellence.

THE GENERAL PICTURE Active Learning Self-Assessment Peer-Instruction and Class Flipping X X

ACTIVE LEARNING ENGAGEMENT EVALUATION LEARNING GAIN

OUTLINE 1. Peer-Instruction & Self-Assessment in an Active Learning Environment 2. Self-Efficacy and Self-Assessment  attainment and confidence levels  objective and subjective measures of confidence 3. Peer-Instruction, Learning Gain & Confidence Gain 4. Student appraisal of the pedagogies 5. Closing the feedback loop 6. Summary of empirical findings.

ETHICAL REMARK You will be presented with data collected during teaching sessions. Students involved have given informed consent for me to analyse their responses and present the results of this analysis. I can assist with ethical queries as well, please ask me.

Active Learning Environment 1. Peer-Instruction & Self-Assessment in an Active Learning Environment

FLIPPED CLASS and PEER-INSTRUCTION Flipped classroom & Peer-Instruction  pre-reading + student interaction  Mazur (1997)  Henderson and Dancy (2009)  well-developed research in Physics and STEM. Learning analytics for Peer-Instruction  Learning gains: Mazour Group - Bates & Galloway (2012)  Student satisfaction: Hernandez Nanclares & Cerezo Menendez (2014). There is not much literature on the links with self-assessment skills Open field, with many unanswered questions  e.g. role of demographics, language, previous background  Pedagogically: self-assessment blends with flipping and Peer-instruction.

SELF-EFFICACY and SELF-ASSESSMENT Academic Self-Efficacy = confidence at performing academic tasks and/or attaining academic goals. Bandura (1977) 1. Mastery of experiences 2. Vicarious experiences 3. Verbal persuasion 4. Environment and settings See also: Pajares (1996) and Ritchie (2015). Idea: Students should develop their self-efficacy to master their learning experience. Measure learning gain along with increased self-efficacy: ‘confidence gain’.

LEARNING GAIN Policy-driven research to assess student learning Arum & Roksa (2010), ‘Academically Adrift’ (US data) OECD approach: assessment of learning outcomes AHELO Project (OECD, 2011 and 2014) US & UK approach: learning gain (‘distance run’ over time) McGrath et al., 2015 HEFCE commissioned research on piloting measures of learning gain Outputs to feed in Teaching Excellence Framework.

ACTIVE LEARNING ENVIRONMENT Introductory Macroeconomics (from 2013 to 2017) year-long module (compulsory 1st year) 250 students (started with 140 in 2013, 250 past 2yrs) 22 lectures (2hrs per week) 8 seminars (every second week, even) 8 workshops (every second week, odd) Students endowed with individual Audience Response Systems (clickers)  continuous data collection facilitated by technology;  comprehensive ethical approval obtained beforehand.

WORKSHOPS – teaching algorithm Round 1 - formative question - 4 choices - no information - no answer Peer-Instruction - students talk - compare answers - explain each other Self-Assessment 2 - confidence question - 4 level Likert-scale - information shared Self-Assessment 1 - confidence question - 4 level Likert-scale - information shared Round 2 - formative question - Identical to R1 - information shared - correct answer

WORKSHOPS – contrast 2 teaching algorithms VES MES Round 1 - formative question - 4 choices - no information - no answer Peer-Instruction - students talk - compare answers - explain each other Self-Assessment 2 - confidence question - 4 level Likert-scale - information shared Self-Assessment 1 - confidence question - 4 level Likert-scale - information shared Round 2 - formative question - Identical to R1 - information shared - correct answer

Holding everything else constant in the world economy: If inflation in a country increases, its… exports decrease and imports stay the same; exports decrease and imports increase; exports increase and imports decrease; imports increase.

“I think I have the skills/knowledge to answer the previous question correctly” strongly agree; agree; disagree; strongly disagree.

Holding everything else constant in the world economy: If inflation in a country increases, its… exports decrease and imports stay the same; exports decrease and imports increase; exports increase and imports decrease; imports increase.

“NOW I think I have the skills/knowledge to answer the previous question correctly” strongly agree; agree; disagree; strongly disagree.

WORKSHOPS – asking the right questions Bloom’s Taxonomy Aim to climb up the pyramid Do not trivialise MCQs. See talk by Simon Lancaster!

DATASETS AND CODING Student Q1 Q2 Q3 …  performance per student confidence by student Formative questions 1 = correct 0 = incorrect Confidence questions 1 = strongly/agree 0 = strongly/disagree 1 0 1 1 2 1 0 0 3 1 1 … …  performance per question confidence by question

PART A – attainment and confidence 2. Self-Efficacy and Self-Assessment Skills PART A – attainment and confidence

WORKSHOPS – data coding For each session (7 in a year in 2013-14, 8 from 2014-15): Code 1st response: 1 = correct 0 = incorrect Code confidence in response: 1 = strongly/agree 0 = strongly/disagree Compute average score and average confidence per student. attainment self-assessment measure measure

WORKSHOPS – data coding For each student (in each session): If student average score > session average score  high-attainment otherwise low-attainment If student average conf. > session average conf.  high-confidence otherwise low-confidence

WORKSHOPS – data analysis Cross-tabulate results:

WORKSHOPS - results What is the relationship between attainment and confidence? No Dunning-Kruger Effect!

WORKSHOP - results What is the relationship between attainment and confidence?

WORKSHOPS – compared with Seminars 7 Seminars running alternated to Workshops. Paper-based assessment – 5-6mins – 4 questions.

WORKSHOPS – compared with Seminars 7 Seminars running alternated to Workshops Paper-based assessment – 5-6mins – 4 questions. In WORKSHOPS  no Dunning-Kruger Effect In SEMINARS  there is Dunning-Kruger Effect.  The modes of delivery for the same type of assessment change the self-assessment outcome.

PART B – confidence and entropy 2. Self-Efficacy and Self-Assessment Skills PART B – confidence and entropy

WORKSHOPS – data coding and modelling For each 1st response question (4-10) across all sessions (7 in a year in 2013-14): Compute entropy index across responses (A,B,C,D)  objective measure of confidence Compute average confidence per question  subjective measure of confidence Model: confidence = f ( entropy ) + session-dummies

WORKSHOPS – results confidence = f ( entropy ) + session-dummies

3. Peer-Instruction Learning Gain & Confidence Gain

OPERATIONALISING TWO GAINS For each 1st and 2nd response to formative assessment questions: % correct R2  % correct R1 Normalised Learning Gain (NLG) = 100%  % correct R1 For each 1st and 2nd response to self-assessment questions: % confident R2  % confident R1 Normalised Confidence Gain (NCG) = 100%  % confident R1

PEDAGOGY EVALUATION STRATEGY Is the pedagogy developing good self-assessment skills? Are students self-assessing correctly over Round 1/2? Is peer-instruction able to generate learning/confidence gain? How does learning/confidence gain relate to initial knowledge/confidence levels (Round 1)? Is learning gain associated to confidence gain? Does the structure of the algorithm affect this relationship? 2016 Vicarious of Experience Scenario (VES) 2017 Mastery of Experience Scenario (MES) only contrasted with VES (4 sessions each).

WORKSHOPS – contrast 2 teaching algorithms VES MES Round 1 - formative question - 4 choices - no information - no answer Peer-Instruction - students talk - compare answers - explain each other Self-Assessment 2 - confidence question - 4 level Likert-scale - information shared Self-Assessment 1 - confidence question - 4 level Likert-scale - information shared Round 2 - formative question - Identical to R1 - information shared - correct answer

RESULT – Self-assessment Skills Round 1 2016 VES =0.429*** R²=0.56 % confident Round 1 2017 VES =0.367*** R²=0.59 2017 MES =0.09*** 0.44 0.33 % correct Round 1

RESULT – Self-assessment Skills Round 2 % confident Round 2 2016 VES =0.282*** R²=0.26 2017 VES =0.322*** R²=0.64 0.60 2017 MES =1.44*** 0.57 % correct Round 2

RESULT – Normalised Gains in 2016 NLG R²=0.39 NCG R²=0.19 0.85 NCG 0.52 NLG 2016 0.37 0.48 % Correct/Confident in Round 1

RESULT – Normalised Gains in 2017 NLG R²=0.21 NCG R²=0.42 =0.132** NCG 0.67 NLG 0.40 0.42 % Correct/Confident in Round 1

RESULT – Learning Gain & Confidence Gain NCG 2016 R²=016 2017 VES =0.19*** R²=0.39 MES =0.125** 0.31 0.28 -1 NLG

REFLECTION The active learning environment discussed today combines two powerful pedagogies: self-assessment and peer-instruction. The role of the teacher does affect confidence levels, which are higher when s/he demonstrates (VES). Alterations of the role played by the teacher in either VES or MES do not compromise the effective interaction between students and the benefits generated by the active learning components of the pedagogy. Students support each other in their learning and develop confidence beliefs along with skills.

4. Student appraisal of the pedagogies

WHAT DO STUDENT THINK? The literature on evaluation of TEL and Peer-Instruction is far too focused on whether students ‘enjoy’ their experience (student satisfaction)  typical of academic practice literature. I want to give more focus on the perception of learning:  1st lecture: introduced the concept of Peer-Instruction asked the students to share what they think of it.  each workshop: asked students to share their view on the session and whether they felt they learnt from each other.  informal end-of-module feedback: what was the most effective component of the blended learning environment mix within the module.

1st lecture: “‘Peer-instruction’ sessions (students teaching each other) are more effective than lectures (teacher teaching students)” No. Respondents = 82

Workshop feedback statement: “I have learnt more Economics by discussing answers with my classmates”

Comparing student opinion about Peer-Instruction as an effective pedagogy before and after exposure

End-of-module Feedback: What is the component of the Macro module which had the strongest impact on your learning?

5. Closing the Feedback Loop

FEEDBACK to STUDENTS During polling sessions:  show results, talk to the students, encourage them;  use comparative-links, show them their progress;  explain what you are doing and why you are doing it. After polling sessions:  share session reports on your VLE and comment on these;  use mail-merge to send individual reports; Go beyond polling:  craft your own perfect pedagogical blend;  be aware of different student needs.

6. Summary of Empirical Findings

SUMMARY of RESULTS In both Round 1&2, confidence levels are lower under MES. However, performance and confidence are consistently positively correlated  students self-assess correctly, irrespectively of MES/VES scenario. Peer-instruction generates higher learning/confidence gain when initial knowledge/confidence in the classroom is neither too high or too low  ‘sweet-spot’ pattern does not depend on MES/VES scenario. Under MES scenario confidence gain is lower, compared to VES scenario. However, confidence gain is consistently positively correlated to learning gain  students develop more confidence as they learn, irrespectively of MES/VES scenario. The pedagogy appears to be robust to teacher intervention  Active Learning!

FINAL REMARKS It took me 6 years to develop this teaching approach & evaluation. Think big….but start small, and build from there. It’s not about the technology, it’s about the pedagogy and, most importantly, it’s about the students. Be concerned about ethics, but do not be discouraged or scared (students are not: JISC, 2016). Choose your demonstrators carefully: your ‘average Fabio’ might be more convincing than a pedagogy expert or a techno-hyper-enthusiast.

STAY IN TOUCH! F.Arico@uea.ac.uk @FabioArico “Promoting Active Learning Through Peer-Instruction and Self-Assessment: A Toolkit to Design, Support and Evaluate Teaching”, Educational Developments, SEDA, 17.1, 15-18.

REFERENCE LIST Aricò, F.R. and Lancaster, S. 2018. Facilitating Active Learning and Enhancing Student Self-Assessment Skills, International Review of Economics Education. https://doi.org/10.1016/j.iree.2018.06.002. Aricò, F., Gillespie, H., Lancaster, S.; Ward, N. and Ylonen, A. 2017. Lesson in Learning Gain: insights and critique, Higher Education Pedagogies, Special Issue on Learning Gain (forthcoming).   Aricò, F. 2016. Promoting Active Learning Through Peer-Instruction and Self-Assessment: A Toolkit to Design, Support and Evaluate Teaching. Educational Developments, SEDA, 17.1, 15-18. Ylonen A. Gillespie H. Green A. 2018. Disciplinary Difference and Other Variations in Assessment Cultures in Higher Education: Exploring Variability and Inconsistencies in One University in England. Assessment and Evaluation in Higher Education, 0(0), 1-9. https://doi.org/10.1080/02602938.2018.1425369.

REFERENCE LIST HEFCE Learning Gain website: http://www.hefce.ac.uk/lt/lg/ Office for Students Learning Gain website: https://www.officeforstudents.org.uk/advice-and-guidance/teaching/learning-gain/ UEA Learning Gain Project website: https://www.uea.ac.uk/learning-gain/home Tweet collection – HEFCE Learning Gain Annual Conference https://wakelet.com/wake/21a75448-6142-4b25-850d-a0b42ec0b67d Fabio Aricò’s Wakelet Tweet curation website: https://wakelet.com/@FArico_Z2lCHBK

Promoting Learning Gain and Confidence Gain in the Classroom Through Self-Assessment and Peer-instruction Dr Fabio R. Aricò @FabioArico ARU – Jun 2018