1. U N I V E R S I T Y O F C A N B E R R A The Role of Assessment in Postgraduate Research Education Dr Margaret Kiley University of Canberra

2. U N I V E R S I T Y O F C A N B E R R A Research  Aimed to contribute to a theoretical dimension of postgraduate ‘pedagogy’  Two main stages of the : l 30+ interviews with very experienced examiners l 26 interviews with inexperienced examiners (< three theses examined) l (Illustrative examples from the analysis of examiners’ reports)

3. U N I V E R S I T Y O F C A N B E R R A Experienced Examiners report that they…  Expect the student to pass as they open the thesis  Are very reluctant to fail a student with most experiencing considerable distress if they do so  Come to a decision about the quality of a PhD by about the end of Chapter 2  Have a formative rather than summative view of thesis examination  Believe that there is risk attached to sending theses to inexperienced examiners  Are reluctant to take much notice of institutional criteria when examining

4. U N I V E R S I T Y O F C A N B E R R A Is there a difference?  Work of Trafford (2003) from 130 vivas it was possible to determine that: l Experienced Examiners tended to ask questions that can be defined as: “Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept” l Inexperienced examiners tended to ask more ‘technical’ questions

5. U N I V E R S I T Y O F C A N B E R R A Trafford’s Categorisation Innovation & Development HIGH Scholarship & Interpretation C. Questions generally relate to issues such as research question, choose of topics, location of study D. Defending doctorateness, contributing to knowledge,, critique of research, synthesizing concept A. Types of questions include resolving research problems, content, structure B. Implications, awareness of, and familiarity with wider literature LOW

6. U N I V E R S I T Y O F C A N B E R R A Purpose of Stage 2 of Research  To explore the contention put forward by many experienced examiners that it was risky to send theses to inexperienced examiners. l Why might this be? l If it is not the case, why does this myth persist? l What are the main similarities between inexperienced and experienced examiners? l What are the main differences between the two groups?

7. U N I V E R S I T Y O F C A N B E R R A Methodology  Interviewed 26 inexperienced examiners recognising two groups: l Those who were inexperienced examiners <3 theses examined and inexperienced supervisors <3 students supervised l Those who were inexperienced examiners <3 theses examined but more experienced supervisors <3 students supervised

8. U N I V E R S I T Y O F C A N B E R R A Methodology (cont)  Interviewed staff from l Canberra l Adelaide l James Cook l QUT l Queensland

9. U N I V E R S I T Y O F C A N B E R R A Interviewees ( )= Previous Study SciMaths/ Eng Soc SciH’tiesTotal Male5 (11)5 (1)3 (4)3 (2)16 (18) Female4 (3)1 (2)3 (5)2 (2)10 (12) Total9 (14)6 (3)6 (9)5 (4)26 (30)

10. U N I V E R S I T Y O F C A N B E R R A Methodology (cont)  Each interview: l Taped, Transcribed and Sent back to interviewee for checking  Analysed independently by two researchers l Searched for themes, although similarities and differences from experienced examiners was not at the fore front of thinking

11. U N I V E R S I T Y O F C A N B E R R A Inexperienced examiners  What is experience/inexperience? Is it related to: l Experience as a supervisor? l Experience as an examiner? l Experience as a researcher?

12. U N I V E R S I T Y O F C A N B E R R A Inexperienced Examiners report…  Have a high level of confidence in ability to examine  Although inexperienced at the doctoral level many had had considerable experience at the Honours level  Adopt a similar approach to the actual process of examining as do their more experienced colleagues  Use criteria similar to experienced examiners  Little in the way of discipline differences  Most very concerned by typographical errors

13. U N I V E R S I T Y O F C A N B E R R A Inexperienced Examiners report…  Surprisingly high percentage wanted to fail their first thesis or said it was ‘awful’  Almost immediate reference to own doctoral experience and advice from their supervisor & seem to have very high expectations of supervisors  More prone, than experienced colleagues, to follow institutional criteria  Tend to focus on the ‘steps’ or components of a PhD rather than the whole  Some felt that they were being examined too  Appreciated feedback from institutions (UQ sends other reports - definite support for that)  Development as an examiner from 1st to 2nd etc

14. U N I V E R S I T Y O F C A N B E R R A Discussion  Suggest that one of their main difficulties is the inability to benchmark  What they report about levels of confidence is not reflected in what they say in response to other questions  See their role as maintaining standards and performing their summative assessment role correctly  Appears that criteria for an acceptable doctorate are even more discipline-related than originally thought (rather than institutionally based)  Is there any truth in the assertion that “Inexperienced supervisors get the ‘dodgey’ students”?

15. U N I V E R S I T Y O F C A N B E R R A From the reports and from comments: A ‘good’ thesis demonstrates…  Critical analysis and argument  Self-critical, rigorous and confident  Contribution to knowledge  Originality, creativity and degree of risk taking  Comprehensiveness & scholarly  Presentation and structure  Sound methodology

16. U N I V E R S I T Y O F C A N B E R R A From the reports and from comments: A less than ideal thesis demonstrates…  Too much detail with lack of analysis  Lack of argument and rigour  Shoddy presentation (typos etc)  Lack of confidence, energy and engagement  Lack of critique of own analysis/sweeping generalisations  Inadequate or poorly expressed methodology and scope

17. U N I V E R S I T Y O F C A N B E R R A Implications for Institutions  Confusion over gradings/ratings and recommendations  How to encourage experienced supervisors to send theses to inexperienced examiners  Providing feedback and support for inexperienced examiners  Greater use of Viva?  Possible use of ‘internal’ examiner in the formative sense

18. U N I V E R S I T Y O F C A N B E R R A Mullins, G. & Kiley, M, (2002).‘It’s a PhD, not a Nobel Prize’: How experienced examiners assess research theses. Studies in Higher Education 27(4) pp. 370-386 Trafford, V. (2003). "Questions in doctoral vivas: View from the inside." Quality Assurance in Education 11(2): 114-122.