1. The feedback conundrum: finding the resource for effective engagement Prof. Margaret Price Director, ASKe Pedagogy Research Centre Faculty of Business http://www.brookes.ac.uk/aske

2. Dynamic Assessment Context Assessment Massification Assessment diversity Holistic approach to assessment Assessment drives learning Transparency and accountability Modularity Purpose of Assessment – various stakeholders Fragmentation of communities Target culture Authentic assessment Uniform Quality Assurance and accreditation systems (Bologna 1999) Employability outcomes Cultural diversity and internationalisation of curriculum Academic Literacies National Student Survey League tables Student engagement

3. Change in HE and effects on assessment Pedagogic developments (e.g. acknowledgement of assessment as key driver of learning, dialogic feedback; discourse of assessment – fairness, cheating, grade inflation; complexity of assessment) Measurement in HE (e.g.league tables, quality processes and measures, fragmentation). Student voice (fees, influence of prior education, student engagement) Market (employability, graduate jobs, authenticity in assessment)

4. Current assessment climate More traditional forms of assessment tend to be taken for granted (Pryor and Croussard 2007) Assessment is being slow to catch up with pedagogic developments Impact of assessment cultures. Teachers and students develop implicit and explicit expectations about learning and assessment (Ecclestone, 2006)

5. Assumption: Designing assessment at module level is sufficient Staff team need a programme view Where there is a greater sense of the holistic programme, students are more likely to achieve the learning outcomes than students on programmes with a more fragmented sense of the programme. (Havnes, 2007) Assessment strategy – looking across a programme o Effects of conflating summative and formative assessment (Price et al 2010, Black and Wiliam, 1998) o Variety in assessment is not always a good thing It is clear how disconcerting students find a large range of assessment activities. It is far better to limit these so students get better at using the feedback to improve performance. (Gibbs, 2011) o Feedback seen as a self contained event (Carless 2011)

6. Assumption: Constructive alignment is sufficient for coherent assessment and feedback Learning activity Assessment Learning outcomes Priority given to assessment? Assessment often only considered superficially in the programme design phase. A well designed programme - assessment and feedback will look after themselves? Implementation – feedback flows Learner

7. Assumption: Assessment standards are straightforward Beliefs are strong that Assessment standards can be made explicit Standards are understood by students Standards are consistent between markers

8. Limitations of explicit articulation Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004). “we can know more than we can tell” (Polanyi, reprinted 1998, p.136). Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987). There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002). Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991).

9. Assessment standards are difficult Assessment judgements rely on local, contextualised interpretations of quality underpinned by tacit understanding of ‘quality’ shared by members of an assessment community (Knight, 2006) A key issue in assessment is that students often do not understand what is a better piece of work and do not understand what is being asked of them particularly in terms of standards and criteria. (O’Donovan et al., 2001)

10. Active student engagement Passive student engagement Informal activities and inputs Formal activities and inputs 1.The Traditional Model Tacit standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Standards explicitly articulated (with limitations) and passively presented to students 3. The Social Constructivist Model Actively engaging students in formal processes to communicate tacit knowledge of standards 4. The ‘Cultivated’ Community of Practice Model. Tacit standards communicated through participation in informal knowledge exchange networks ‘seeded’ by specific activities. The Past The Future O’Donovan, Price & Rust 2008

11. Assessment judgements are complicated –Markers rank and measure standards simultaneously (Wolf 1997) –Holistic judgements are justified by criteria (Bloxham and Boyd 2011) – –Markers apply different interpretations of key words and phrases within written ‘standards’ (Saunders and Davis 1998; Ecclestone,2001, Webster et al., 2000, (Price & Rust 1999) ) –Markers subconsciously use other criteria (Price 2005) –Markers can use only a small number of criteria at any one time (Elander 2002) –Markers allocate their own weightings to criteria (Ecclestone 2001) –Markers may be harsher if they have more time and/or have to justify their judgements (Baume et al 2004)

12. Assumption: There is a common view about feedback

13. Impact on feedback? There has been quite a bit of research on feedback… Don’t read it (Hounsell,1987; Gibbs and Simpson, 2002) Too vague (Higgins, 2000; Walker, 2009) Not understood (Lea and Street, 1998; Weaver 2006; Sadler 2010) Subject to interpretation (Ridsdale, 2003; Orsmond and Merry, 2011) Unidirectional (Nicol 2010) Damages self-efficacy (Wotjas, 1998) Importance of the ‘relational’ (Price et al. 2010)

14. The aim of the project: to understand better what influences whether feedback is viewed as ‘good’ or ‘bad’ by students (and in NSS scores). What makes good feedback good? An ASKe Pedagogy Research Centre project in collaboration with Cardiff, funded by the HEA

15. Domains pertaining to the feedback itself (the traditional focus of advice to improve feedback)  Technical factors – presentation and content – legibility, interpretatability (incomprehensible ticks/remarks), levels of explanation.  Particularity of feedback – evidence of engagement with particular piece of work, personalisation valued over ‘standardised criterion-based feedback  Recognition of student effort – evidence of time spent by markers as well as supportive detail.

16. Domains relating to the context of feedback  Assessment design - clarity of purpose, relevance, realistic (e.g word count, time)  Feedback pre-conditions – clear criteria, task, instructions… dialogue.  Marker predictability – particularly if the marker is not the person who had briefed them or provided formative feedback. Sites of Silence  Timing  Design of assessment patterns

17. Domains pertaining to the development & expectations of the student  Student mark expectations – some influence but not overwhelming. Effort often equated with marks and influences the type of feedback seen as useful.

19. Domains pertaining to the development & expectations of the student  Student mark expectations – some influence but not overwhelming. Effort often equated with marks and influences the type of feedback seen as useful.  Student epistemology, resilience and beliefs –dualistic students and model answers and specificity; –intrinsic (learning) and extrinsic (mark) motivations. –poor self evalaution and reliance on feedback –criticism, critique, transaction or conversation

20. Summary of findings  You don’t need to get it all right all the time  Domains overlap, compensate each other, have strong interrelationships, are not mutually exclusive.  Domains of influence are not causal, prioritised, they are context dependent and influences outside the ‘feedback’ artefact are important.  Student perceptions of feedback are shaped by: Some aspects of the feedback itself – a necessary but insufficient condition for feedback being seen as good Pre-feedback conditions Qualities and perspectives of the student

21. Where do we go from here? ‘ The domain that perhaps offers greatest unexploited scope for improvement concerns student learning development. Successful students use feedback differently and more effectively (without the context or the feedback changing) and it is possible to change how students perceive feedback and what they do with it.’ (Research report available at https://www.brookes.ac.uk/aske/)

22. So…. a long term project Look beyond the feedback artefact itself. An overemphasis on technical factors at the expense of contextual elements such as good teacher student relationships can be detrimental. Take a Programme level focus Create resource through development of student assessment literacy Independent learners will cope better with the imprecise nature of assessment and be able to engage with feedback Develop a discourse around assessment and feedback