Lin Norton, Bill Norton, Lee Shannon and Frances Phillips Assessment design, pedagogy and practice: What do ‘new’ lecturers think? Lin Norton, Bill Norton, Lee Shannon and Frances Phillips 24 October 2009 ISSOTL 2009

Acknowledgements Research has been funded by the Write Now CETL The researchers include: Ola Aiyegbayo (formerly of Liverpool Hope University) James Elander (Derby University) Katherine Harrington (London Metropolitan University) Frances Phillips (Liverpool Hope University) Peter Reddy (Aston University) Lee Shannon (Liverpool Hope University) 24 October 2009 ISSOTL 2009

Background: Is there a problem with assessment? Authentic assessment can be defined as assessment that is pedagogically appropriate- it frames students’ views of HE, it has a major influence on their learning and it directs their attention to what is important (Boud & Falchikov, 2007) But… In the USA, many institutions tend to adopt assessment approaches more traditional than those recommended in the literature (Peterson & Einarson, 2001) In the UK context, Rust (2007) asserts that many assessment practices are poor and not underpinned by pedagogical literature. There seems to be a gap between SOTL and what it has to say about assessment and actual assessment practice 24 October 2009 ISSOTL 2009

So what is currently happening in HE? Widely held view that assessment should be for rather than of learning (Black & Wiliam, 1998; Assessment for Learning: AfLCETL http://www.northumbria.ac.uk/cetl_afl/) But… is this what lecturers(faculty) think- particularly new lecturers, and, if so, are they able to put their beliefs into practice? And… what part do taught programmes for lecturers have to play? 24 October 2009 ISSOTL 2009

What part does a university teaching programme play? An initial interview study was carried out with 10 new lecturers taking a university teaching course (Norton, Aiyegbayo, Harrington Elander & Reddy, 2010) Our findings suggested that the course was generally perceived as useful and influenced the way that the new lecturers thought about assessment: ‘The assessment module [in the course] revealed to me that assessment is not just about assessing how much people know but should be used as a tool in learning. This was an eye opener for me.’ (G) 24 October 2009 ISSOTL 2009

Interview study cont.. When asked if they could change assessment practice easily, 6 said no and 4 said yes but with reservations: Not really because it is set in stone in the module proposal. You have to jump through many hoops if you are going to change the assessment techniques.’ (A) ‘No! I get the impression that they are set in stone…. I think that hurdles of going to various panels to have your module changed puts people off…I get the impression from talking to colleagues that the process is long-winded and bureaucratic.’ (C) We concluded that what new lecturers learn about assessment does not always readily translate into practice. This may be caused by a complex interaction of institutional, departmental and individual factors (Becher & Trowler, 2001; Fanghanel, 2007), which we have called ‘constraints’ We wanted to explore these constraints further, using a questionnaire approach 24 October 2009 ISSOTL 2009

Current research study First of several large scale quantitative studies carried out by the Write Now CETL research team investigating lecturers’ views on assessment, marking & feedback. This study focuses on lecturers who were taking, or had recently taken, a university teaching course and their views about assessment design. Aims To analyse whether there were any potential constraining factors hindering lecturers from putting what they had learned about assessment into practice. To develop the Assessment Design Inventory (ADI) as an instrument 24 October 2009 ISSOTL 2009

Development of the ADI The Assessment Design Inventory (ADI) comprises 40 items developed from over 600 items constructed from 84 interviews with lecturers (new and experienced) from 18 disciplines, in 5 institutions. An earlier version of the ADI consisting of 68 items was piloted on a sample of 29 lecturers from one UK university The ADI was sent out as an online survey aimed at those lecturers currently on HE teaching courses or who had recently (in the last 2 years) completed such a course. Access to respondents was by invitation via HE teaching course leaders at all UK institutions (150). Around half the institutions agreed to circulate the invitation to their course participants 24 October 2009 ISSOTL 2009

Participants 586 fully completed inventories ( 325 females, 256 males, & 5 not specified ) 80 discipline areas and 66 institutions from England, Ireland, Scotland & Wales 276 were currently on a HE Teaching course; 308 had already attained an HE Teaching qualification. University teaching experience ranged from less than a year to 26 years: Years Under 2 2 to 5 6 to 10 11+ N 232 197 99 49 24 October 2009 ISSOTL 2009

Assessment Design Inventory(ADI) Initial exploratory analysis suggests 3 underlying factors: Constraints e.g. ‘Changes to my assessment design are hindered by external factors (e.g. cost, high student numbers, time).’ Desirable practice e.g. ‘Involving students in the assessment design would encourage them to engage in the assessment task’. Purpose e.g. ‘The main aim of assessment is to document student achievement to employers.’ 24 October 2009 ISSOTL 2009

Constraints Results will be presented for the 12 items on the ADI that specifically asked about: the potential constraints on turning assessment design philosophy into assessment practice We were particularly interested in looking for any effects on responses relating to: participation on HE teaching courses length of experience in teaching in Higher Education discipline, gender, institution 24 October 2009 ISSOTL 2009

The effect of the University teaching programme on new lecturers’ beliefs about assessment My experience on the University teaching programme has changed my views on assessment practice. 74% agree 16% disagree 10% uncertain [Mean=3.80, SD=0.99] No effect for teaching qualification or length of experience 24 October 2009 ISSOTL 2009

How do new lecturers feel about innovating assessment? (1) New assessment methods are needed to improve current practice. 69% agree 17% disagree 14% uncertain [Mean=3.67,SD=0.95] Lecturers who had already attained a teaching qualification agreed more strongly than those currently taking a teaching course There is little incentive for lecturers to innovate in their assessment practice. 61% agree 32% disagree 7% uncertain [Mean=3.40,SD=1.18] Lecturers with 11+ years experience were less likely than those with less than 2 years experience to agree that there was little incentive to innovate 24 October 2009 ISSOTL 2009

How do new lecturers feel about innovating assessment? (2) Innovation in assessment is not well received by students. 31% agree 52% disagree 16% uncertain [Mean=2.76,SD=1.15] Lecturers with 11+ years experience were more likely than those with less than 2 years experience to disagree. I feel my ideas are valued when developing assessments. 65% agree 16% disagree 20% uncertain [Mean=3.53,SD=0.92] Lecturers with 11+ years experience agreed more strongly than those with less than 2 years experience. In order to introduce innovative assessment I would consider avoiding official procedures. 27% agree 57% disagree 16% uncertain [Mean=2.64,SD=1.09] No effect for teaching qualification or length of experience 24 October 2009 ISSOTL 2009

What do new lecturers feel are external constraints? Changes to my assessment design are sometimes hindered by external factors (e.g. cost, high student numbers, time). 75% agree 20% disagree 4% uncertain [Mean=3.81,SD=1.11] QAA requirements seldom allow leeway in assessment design. 29% agree 41% disagree 31% uncertain [Mean=2.91,SD=0.95] If external constraints (e.g. QAA, subject benchmarks) were removed, I would be more willing to change my assessment practice. 35% agree 41% disagree 24% uncertain [Mean=2.98,SD=1.10] No effect for teaching qualification or length of experience for these 3 items 24 October 2009 ISSOTL 2009

What do new lecturers think about students when designing assessment It is possible for students to ‘go through the motions’ to satisfy assessment requirements without learning anything. 57% agree 39% disagree 5% uncertain [Mean=3.22,SD=1.25] No effect for teaching qualification or length of experience It is a challenge to design assessments for students of differing levels of commitment. 77% agree 20% disagree 4% uncertain [Mean=3.78,SD=1.09] I find it a challenge to design pedagogically sound assessment because of student focus on grades. 57% agree 35% disagree 10% uncertain [Mean=3.27,SD=1.09] Lecturers with 11+ years experience found this less of a challenge than those with 5 years or less experience. 24 October 2009 ISSOTL 2009

What do new lecturers think about students when designing assessment I find it a challenge to design assignments for students of differing levels of ability. 61% agree 35% disagree 5% uncertain [Mean=3.32,SD=1.09] Lecturers with 11+ years experience found this less of a challenge than those with less than 2 years experience. 24 October 2009 ISSOTL 2009

Conclusions We are still at the early stages in our analysis but our interim findings suggest that: Lecturers (both inexperienced and experienced) taking a university teaching course do think that it has changed their views on assessment. Over three quarters of our respondents thought that assessment needs improving The fact that there were some significant differences in responses between the experienced lecturers and those with less experience suggest that the former may be more likely to find their way around the various constraints when designing their assessments 24 October 2009 ISSOTL 2009

Further research Further analysis of our findings from this study is being carried out and will inform our other work on developing a marking and feedback inventory (currently under construction) The ADI itself is still under development and has been made freely available as a research tool for all on the Write Now website http://www.writenow.ac.uk 24 October 2009 ISSOTL 2009

References Becher, T. & Trowler, P. (2001) Academic tribes and territories: Intellectual Enquiry and the Culture of Disciplines 2nd edition Buckingham: Society for Research in Higher Education and the Open University Press. Black, P. and Wiliam, D. (1998) Assessment and Classroom Learning. Assessment in Education 5 (1) 7-74 Boud, D & Falchikov,N. (2007) Rethinking assessment in higher education. Learning for the longer term, Routledge CETL: Assessment for learning (AfL) http://www.northumbria.ac.uk/cetl_afl/whatis/ Fanghanel, J. (2007) ‘Investigating university lecturers’ pedagogical constructs in the working context’. York: Higher Education Academy. http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/fanghanel.pdf Norton, L, Aiyegbayo, O. , Harrington, K. , Elander, J. and Reddy, P. (2010) 'New lecturers' beliefs about learning, teaching and assessment in higher education: the role of the PGCLTHE programme', Innovations in Education and Teaching International, 47: 4, 345 — 356. Peterson, M.W. & Einarson, M.K. (2001) What are colleges doing about student assessment? Does it make a difference? The Journal of Higher Education, 72,6, 629-669 Rust, C (2007) "Towards a scholarship of assessment" Assessment & Evaluation in Higher Education 32, 2, 229-237 24 October 2009 ISSOTL 2009