1. Oral Assessment in an Advanced Lab Course 2009 Topical Conference on Advanced Laboratories University of Michigan, Ann Arbor 23-25 July 2009

2. 25 July 2009David Bailey - Oral Assessment2 What’s the point?

3. 25 July 2009David Bailey - Oral Assessment3 Not about Oral Presentations i.e. not about student talks At Toronto students give oral presentations in –2nd Year Lab –Research & Reading Courses Not in Advanced Lab

4. 25 July 2009David Bailey - Oral Assessment4 Oral Assessment Assess –Lab accomplishments & knowledge –Oral communication skills Assist instructor understanding of –notebook –lab work

5. 25 July 2009David Bailey - Oral Assessment5 Communications vs Physics Need to decide if marks will be allocated for communication skills –Explicitly A specific part of marking scheme –Indirectly Lousy communicators loose marks because examiners can’t figure out what the student knows or accomplished.

6. 25 July 2009David Bailey - Oral Assessment6 Toronto Grading Scheme Simulated Data Analysis 2 % 3 Experiments 60 % –3/10 from interview Formal Report18 % Final Oral20 % Total Oral Component38% About 1/3 of this mark is communications skill, i.e.10%

7. 25 July 2009David Bailey - Oral Assessment7 Benefits & Drawbacks Positives –efficient and flexible Negatives –stressful –perceived to be more subjective than marking notebooks.

8. 25 July 2009David Bailey - Oral Assessment8 Oral Examinations “a stressful method of student assessment with poor objectivity and reliability and doubtful validity” –P. T. Jayawickramarajah, “ Oral examinations in medical education ”, Medical Education 19 (1985) 290-293.

9. 25 July 2009David Bailey - Oral Assessment9 Complaints This past year, for the first time we got complaints from a few students that some professors were unfair and too tough in their interviews. They wanted the whole mark based on the notebook.

10. 25 July 2009David Bailey - Oral Assessment10 Efficient Questioning Based on Fig. 5 from J. R. Platt, “ On Maximizing the Information Obtained from Science Examinations, Written and Oral ”, Am. J. Phys. 29 (1961) 111-122. Weak Student Strong Student Knowledge Ignorance Efficient D, E Inefficient A, B, C

11. 25 July 2009David Bailey - Oral Assessment11 Assessment Format Directed Discussion i.e. We ask questions First question is usually variant of “What is the point?” At least 1 question each –about hardware –that requires notebook –about physics open (note)book

12. 25 July 2009David Bailey - Oral Assessment12 Experiment Mark ~ 1/3 each on –student’s work in lab –student notebook –interview but components are strongly interdependent.

13. 25 July 2009David Bailey - Oral Assessment13 Experiment Interview 20 minutes long –formerly 1 hour (or more) 1 Examiner –Professor supervising experiment who has already “read” notebook Discussion –clarifies understanding of notebook and what was done in lab –provides immediate feedback

14. 25 July 2009David Bailey - Oral Assessment14 Final Oral Exam 25 minutes long –5 minutes between exams 3 examiners –2 Professors and 1 TA try to have 1 expert and 1 novice 2 experiments are discussed – student chooses 1st open (note)book

15. 25 July 2009David Bailey - Oral Assessment15 Oral Exam Marks After student leaves, –each examiner writes down a mark in secret without any discussion –take average May then be a minute of discussion if the marks vary widely –does not change the student’s mark –for examiner education only

16. 25 July 2009David Bailey - Oral Assessment16 Bias and Accuracy Bias –Unavoidable –Be self-aware –Use multiple assessors Accuracy –Notebooks can be revisited, but not clear marks are more accurate “After 4 decades, I can now just about tell an A notebook from a B notebook” David H., U. Toronto

17. 25 July 2009David Bailey - Oral Assessment17 F 50 60 70 80 90 Grades Students 40% 20% 0% Quantum Mech. Advanced Lab Classical Mech. Grade Distributions Averages 71±0.9 68±0.6 80±0.6

18. 25 July 2009David Bailey - Oral Assessment18 Marking Correlations (Correlation Coefficients) 100 80 60 40 40 50 80 100 Oral vs Experiments (0.61±0.03) X Quantum vs Classical (0.68±0.03) Oral vs Classical (0.61±0.04) Oral vs Quantum (0.50±0.04)

19. 25 July 2009David Bailey - Oral Assessment19 Marking Correlations (Correlation Coefficients) 100 80 60 40 40 50 80 100 Oral vs Experiments (0.61±0.03) X Quantum vs Classical (0.68±0.03) Expt vs Classical (0.64±0.04) Expt vs Quantum (0.43±0.05)

20. 25 July 2009David Bailey - Oral Assessment20 Examiner Variations Oral examiner averages vary little 77-81 Slightly more variation in spread 8-11 But marks are well correlated >0.76

21. 25 July 2009David Bailey - Oral Assessment21 Marking Spreads Expect smaller mark differences for Orals than for Experiments, where both both in marker and student work varies. Average RMS spread –by different examiners for same student on same Oral Exam 4.1±0.1 –for different experiments by same student 7.2±0.3

22. 25 July 2009David Bailey - Oral Assessment22 Summary Oral Interviews and Examinations –powerful tools for assessing student competence and achievement. –relatively efficient and flexible but –stressful –perceived to be more subjective and inconsistent Toronto’s experience has been good.