1. 6 July 2011Assessment in Higher Education Conference1 The assessment, marking and feedback inventory (AMFI): a tool for exploring the practitioner perspective Lin Norton, Bill Norton, Kamel Mansi

2. 6 July 2011 Assessment in Higher Education Conference 2 Acknowledgments: our thanks to all the lecturers at Liverpool Hope who helped with this research

3. 6 July 2011 Assessment in Higher Education Conference 3 Research Background This earlier research only tells part of the story in two respects: 1) it concentrated on ‘new’ lecturers 2) it only looked at their views on assessment design The current study was designed to look more widely at the whole issue of assessment, marking and feedback with lecturers with a range of experience all working at Liverpool Hope University Previous research carried out by the Write Now CETL research team focused on newly qualified lecturers’ views about assessment design. We found that certain ‘constraints’ could hinder sound assessment design being put into practice (Norton, Norton & Shannon, in press).

4. 6 July 2011 Assessment in Higher Education Conference 4 Research questions 1.What do lecturers think about assessment methods, marking and feedback practices? 2.Are there any differences between the subject disciplines ? 3.Can a questionnaire on assessment, marking and feedback be developed that would be useful for both research and practitioner purposes?

5. 6 July 2011 Assessment in Higher Education Conference 5 Theoretical background: the effects of the discipline Neumann Parry & Becher (2002) used Biglan’s (1973) original classification of hard- soft and pure –applied and found differences in teaching and research:  hard (pure and applied) - strongly committed to research and less committed to teaching, (generally seen as relatively straightforward and unproblematic), collaborative research and teaching  soft (pure and applied) - greater emphasis on scholarly knowledge that translates readily into teaching, more emphasis on individualistic enquiry and not so much acceptance of joint teaching

6. 6 July 2011 Assessment in Higher Education Conference 6 But what about assessment… ?? Since assessment is so closely aligned to learning and teaching, might there be similar differences in lecturers’ assessment, marking and feedback practices and attitudes, related to their subject disciplines ?? White & Liccardi (2006) drew some distinctions between the disciplines and assessment methods favoured http://eprints.ecs.soton.ac.uk/12576/2/H arnessing_Insight_into_Disciplinary_Diff erences_to.ppt http://eprints.ecs.soton.ac.uk/12576/2/H arnessing_Insight_into_Disciplinary_Diff erences_to.ppt DisciplineFavoured assessments Hard pure (Natural Sciences) Specific & focused exam questions Objective tests (quantitative nature of knowledge) Hard applied (Engineering ) Preference for exam questions, specifically problem-solving Soft Pure (Social Sciences & Humanities) Essay questions, short answer questions, oral presentations, continuous assessment Soft Applied (Nursing, Education) Essays, project-based assignments, peer & self assessments

7. 6 July 2011 Assessment in Higher Education Conference 7 The concept of signature pedagogies Lee Shulman, former president of the Carnegie Foundation for the Advancement of Teaching 10 year study to understand how people are prepared for practice in law, engineering, the clergy, teaching, nursing and medicine. Argued that education for professionals has to include more than knowing, and that understanding the discipline is only part of what we should be teaching Signature pedagogies are not stable and unchanging and are as likely to change as professional practices themselves

8. 6 July 2011 Assessment in Higher Education Conference 8 Characteristics of the signature pedagogies Shulman described a mode of teaching that has become inextricably identified with preparing people for a particular profession, he says there are three defining characteristics: 1.“… it’s distinctive in that profession… 2.… it is pervasive within the curriculum. So that students learn that as they go from course to course, there are certain continuities that thread through the program that are part of what it means to learn to “think like a lawyer,” or “think like a physician,” or “think like a priest.”” 3.(it) … cuts across institutions and not only courses. Signature pedagogies have become essential to general pedagogy of an entire profession, as elements of instruction and of socialization.

9. 6 July 2011 Assessment in Higher Education Conference 9 So where is assessment in signature pedagogies? Shulman writes about teaching processes but appears silent on the subject of assessment processes Taking the very broad concept of signature pedagogies as a proxy for teaching process differences, then there presumably must be assessment process differences… In our work on assessment design (Norton et al, in press) we found some discipline differences on:  ‘desirable assessment practice’ factor lecturers from soft applied disciplines scored higher on the overall total than lecturers from soft pure disciplines, hard pure disciplines and hard applied disciplines  ‘constraints to desirable practice ’ factor lecturers from hard pure and hard applied disciplines scored higher on the overall total than lecturers from soft pure and soft applied

10. 6 July 2011 Assessment in Higher Education Conference 10 Development of the Assessment, Marking and Feedback Inventory (AMFI) A pilot version of the AMFI was developed by Dr Kamel Mansi (former Write Now CETL postdoctoral researcher) This consisted of four main sections (The items were derived from over 80 interviews with lecturers from 5 UK institutions, as well as from the literature) : 1.methods of assessment (52 item checklist from Bloxham & Boyd (2007)) 2.rationale for choice of assessment method (13 statements) 3.marking attitudes and practices (22 statements) 4.feedback attitudes and practices (31 statements) The AMFI was made available online to Hope lecturers. 45 completed inventories from 17 disciplines.

11. 6 July 2011 Assessment in Higher Education Conference 11 Can signature pedagogies be applied to subjects studied at Hope? We thought it could, given the strong government agenda that university education should be linked to employability skills BUT Hope is mainly liberal arts; nevertheless we thought subjects that could be counted as professionally-orientated might include  Education  Music, Drama, Fine Arts  Business studies, Marketing  Psychology, Sociology, Health So we were keen to analyse our AMFI results to see if there were any subject differences, but…

12. 6 July 2011 Assessment in Higher Education Conference 12 …not enough participants Out of the completed inventories, there were not sufficient numbers to make any meaningful comparisons throughout, as many of the subjects were represented by one or two lecturers. In subjects where there were more, such as Business (N=5), not all of them completed all the questions. This left us with the option of assigning the subjects to Biglan’s (1973) broad categories of hard applied, hard pure, soft applied, and soft pure.

13. 6 July 2011 Assessment in Higher Education Conference 13 Which subject disciplines did the participating lecturers represent? HARD APPLIED (5) Computing Health SOFT PURE (12) English Geography Music Psychology Sociology Theology & Religious Studies SOFT APPLIED (24) Arts Business Studies Childhood & Youth Studies Disability Studies Drama & Theatre Studies Education Marketing Media Sport HARD PURE No lecturers

14. 6 July 2011 Assessment in Higher Education Conference 14 Section 1. Methods of assessment 52 methods presented and participants were asked to look at each one and respond:  Have used  Would like to use  Not Familiar See handout

15. 6 July 2011 Assessment in Higher Education Conference 15 Assessment methods continued Of Bloxham & Boyd’s 52 methods of assessment presented, only one (patchwork texts) had not been used by any of the respondents, although 3 lecturers said they would like to use this method 19 of the 52 methods were unfamiliar to over a third of our lecturers. Many respondents expressed interest in using methods of assessment they did not currently use..

16. 6 July 2011 Assessment in Higher Education Conference 16 Most popular assessment methods respondents would like to use MethodN=45 Electronic presentation (web pages)38% Writing abstracts31% Debate speech29% Film or radio programmes29% Simulation exercises24% Computer based assignment22% Annotated bibliography22% Fieldwork reports20% Review of book, article, website, etc.20% Placement reports20%

17. 6 July 2011 Assessment in Higher Education Conference 17 Most popular assessment methods used MethodN=45 Essay98% Dissertation93% Research project93% Presentation91% Reflective learning assignments80% In-class tests76% Examinations (unseen, seen, open-book, case study, take-away) 75% Portfolio (written)71%

18. 6 July 2011 Assessment in Higher Education Conference 18 Differences between the disciplines No real differences in the most commonly used assessment methods but there was a small difference in the range of methods. Hard applied lecturers had used 41 methods Soft pure lecturers had used 47 methods Soft applied lecturers had used 46 methods

19. 6 July 2011 Assessment in Higher Education Conference 19 Section 2: Rationale for choice of assessment method 13 items were presented, e.g.  Takes into account students’ preferences  Prepares students for employment Participants were asked to rate how important they felt each item was:  Very important  Important  Not important  Completely unimportant

20. 6 July 2011 Assessment in Higher Education Conference 20 Most important rationales for choice of assessment Very important/important Overall (N=45) Provides a sample of what students are able to do with the knowledge they have acquired98% Develops students’ writing skills97% Prepares students for employment90% *Prevents students from plagiarising89% Allows students to develop oral as well as written skills86% * Preventing students from plagiarising is not a pedagogical rationale, but is commonly cited for using specific methods of assessment such as portfolios (Irons, 2004) and exams (Norton et al, 2006)

21. 6 July 2011 Assessment in Higher Education Conference 21 Section 3: Marking attitudes and practices This section contained 22 items e.g.  The external examiner system does not ensure grading fairness (attitude)  I have learned to mark myself without any professional training (practice) Participants were asked to respond to each one by choosing one of the following responses:  Strongly agree  Agree  Disagree  Strongly disagree

22. 6 July 2011 Assessment in Higher Education Conference 22 Section 3. Are there any differences between the subject disciplines in what lecturers think about marking? Hard applied lecturers believed more strongly than soft pure lecturers that professional training in marking was not needed Hard applied lecturers agreed more strongly than both soft pure and soft applied lecturers that giving students highly detailed guidelines increased the likelihood of students challenging their grades

23. 6 July 2011 Assessment in Higher Education Conference 23 Section 4: Feedback attitudes and practices This section of the AMFI contained 31 statements e.g.  Face to face communication with students is the best way to give feedback (attitude)  I seek advice from colleagues for providing clear information to students (practice) Participants were asked to respond to each one by choosing one of the following responses:  Strongly agree  Agree  Disagree  Strongly disagree

24. 6 July 2011 Assessment in Higher Education Conference 24 Section 4. Are there any differences between the subject disciplines in what lecturers think about feedback? Hard applied and soft applied lecturers agree more strongly than soft pure lecturers that giving feedback puts an unfair workload on lecturers Hard applied and soft applied lecturers agree more strongly than soft pure lecturers that feedback sessions in groups were more effective than individual feedback Hard applied lecturers were more likely than soft applied and soft pure lecturers to agree that lack of time prevented them from giving good feedback

25. 6 July 2011 Assessment in Higher Education Conference 25 Is there a pattern? Too small a data set to answer with any certainty, and out of all the items, we only found 5 that were statistically different. Some indications that hard applied lecturers were ‘more traditional’ in their approach to marking and feedback, they were less keen to spend time on it and used fewer assessment methods than lecturers from the other two disciplines This lends some support to our earlier finding which showed hard applied lecturers were more likely to be constrained in their assessment design.

26. 6 July 2011 Assessment in Higher Education Conference 26 Developing the AMFI as a research tool and as a practitioner tool 1.Research: Currently refining the items (correlational and reliability analyses). Would like to try it out on a much larger sample to see if signature pedagogies exist in assessment, and if so what are the implications for engendering change? 2.Practice: Several participants commented that the exercise of completing the AMFI had helped them think more closely about their assessment, marking and feedback practices and attitudes, so it could be used in a similar way to the Assessment Design Inventory as a trigger for debate and discussion ( within departments and/or on university teaching programmes)

27. 6 July 2011 Assessment in Higher Education Conference 27 A final question… How easy is it to change assessment in Higher Education? (…if it needs changing) in a sector where generic assessment pedagogy can sometimes be seen as irrelevant to:  Departments (Becher & Trowler, 2001; Knight & Trowler,2000; Fanghanel, 2007)  Disciplines (Biglan,1973; Neumann et al, 2002)  Professions (Shulman, 2005)