1. Using exemplars to develop students’ assessment and feedback literacy
Professor Kay Sambell
Assessment Fiesta, Anglia Ruskin University,
September 2014

2. What I plan to do today
To consider some of the roles exemplars might play in helping students to better understand university assessment and feedback processes
to help us identify some issues to think about if we use exemplars with our students
Use one case study as a concrete example to discuss, drawing attention to questions about
Design features: what we think of when selecting effective exemplars
Process features: what we do when we put them into practice with our students

3. The literature increasingly refers to the use of exemplars
To help students develop a deeper understanding of assessment expectations and achieve transparency
Help students self-assess by providing concrete illustrations of assessment standards and criteria (see, particularly, the work of ASKe, Oxford Brookes)
To ‘scaffold’ students’ learning and ‘load-up’ or ‘flip’ feedback by integrating it with classroom teaching
feedback is ‘any information a teacher (or peer) can provide that helps a student understand how they could have completed the task in a better way to produce work at a higher level of quality’ (Hendry, 2013; Hounsell, forthcoming)

4. …indispensable conditions….
The student comes to hold a concept of quality roughly similar to that held by the teacher
Is able to monitor continuously the quality of what is being produced during the act of production itself
Has a repertoire of alternative moves or strategies from which to draw
(Sadler 1989)

5. Challenges inherent in the student experience of feedback: some pointers from research
‘Gap’ between staff and students’ assessment and feedback literacies (Price et al, 2012)
Students don’t understand assessment criteria/standards/what constitutes good work (i.e. the context in which comments are framed), so can’t fully interpret it/perceive its intended meaning or ‘decode’ it.
Students often have limited models of feedback
Don’t appreciate learners’ role in the feedback process (Price et al, 2011)
Directive teachers’ comments on coursework at school ‘telling’ pupils what to put in/take out (Sambell, 2011)
Timing
Too late (Hounsell, 2007)
Can’t perceive links to future work (Handley & Williams, 2011)
Format: ‘feedback-as-telling’
Monologic: one way street (Nicol, 2010; van de Ridder, 2008)
Don’t understand assessment discourse e.g. ‘descriptive’, ‘analysis’

6. How do we address the deeper level problem of developing students’ appreciation of what’s required?
Simply publishing assessment rubrics is rarely enough....
‘Transparency’ seems to have become muddled with the idea of ‘writing things down’ (Orr, 2007: 646)
‘Students need to learn about assessment in the same way they do about anything else’ (Rust et al, 2005).
active, participative, dialogic experiences (based on social-constructivist approaches)
Ie assessment is a social practice, learned slowly by participation

7. ….we need to find effective ways of making our (often tacit) assumptions visible and accessible
It is difficult to communicate the meaning of assessment criteria.
They often include words such as ‘analytical’, ‘critical’ or ‘satisfactory’ which are relatively meaningless unless you have a framework in which to understand them (Bloxham, 2009).
Marking rubrics are too abstract: students need to ‘see’ and ‘feel’ what they look like in concrete productions within the discipline (Sambell, 2011; Hendry et al, 2012)
Research has shown that assessment tasks in HE are implicitly framed by the specific expectations, cultural and disciplinary orientations of the lecturers who design them (Bloxham & West, 2007; Lillis, 2001; Lea and Street, 1998).

8. Exemplars: a useful way forward in ‘induction’/development of assessment literacy?
Exemplars are ‘key examples chosen so as to be typical of designated levels of quality or competence. The exemplars are not standards themselves but are indicative of them. ..they specify standards implicitly.’
(Sadler, 1989 cited in Handley et al, 2008, p44).

9. Putting exemplars into practice

10. Design features: Exemplars can take different forms…
Handley et al (2008: 44) suggest that exemplars
may be complete assignments or excerpts
may be authentic pieces of student work, or may be (re)constructed by staff (so as to illustrate specific pedagogic points in as transparent manner as possible)
may be annotated with feedback to
help students understand what tutors look for
help students build their self assessment skills

11. Published studies have emphasised that active engagement and dialogue are crucial
Have tended to use exemplars when the genre or format of summative assessment is ‘unusual’ in some way
Orsmond (2002)
Scientific posters in Biology
Hendry et al, 2011
Legal letters of advice to clients in Law
Essay writing with Mathematicians
Bell et al (2013)
Exemplars of extracts from reflective reports, annotated with tutor comments and combined with marking guides and grade descriptors

12. An example of exemplars from the literature (Bell et al, 2013)

15. Today’s case study

16. In the case study…
Exemplars were based on short pieces of informal student writing (prior students’ work) explaining concept in the discipline [Childhood Studies]

17. Process features: How were exemplars used in the case study?
Preparation before workshop
Students prepared short piece of writing (>1 side A4).
500 word response to the task ‘Explain the social construction of childhood to a 1st year student.’
Phase 1
Brought this to session, where, after discussing/generating criteria, they were initially given 3 exemplars to read and discuss.
Students asked to work in small groups and place exemplars in rank order, and start discussing some feed forward comments
Phase 2
Tutors revealed and discussed rationale for their rankings.
Phase 3
Students asked to generate feedback for each exemplar& reflect on how, in the light of the tutor dialogue, they would change own work.

18. Workshop activity
Please have a go at ranking the exemplars as if you were in the students’ shoes (phase 1)
What sort of feedback comments emerge amongst the group?
How confident are you that you would agree
Amongst yourselves?
With the subject lecturers teaching the course?

19. Some possible discussion points
When you chatted about the exemplars with peers, what did you notice and talk about?
Visible criteria? (use of language, spelling, grammar, citation issues)
Invisible criteria? (relevance, use of sources)

20. Lecturers’ perspectives on design features
Chose 3 ‘actual’ exemplars carefully to
‘Engineer in’ the essence of the discipline and illuminate effective and ineffective approaches.
Represent ‘common mistakes novices make’, to bring them to visibility, trying to help students notice conceptual mistakes in time to change approaches, if necessary.

21. They explained their grades/reasons…
All exemplars displayed errors of convention (spelling, citation, grammar).
The lecturers saw these features as of less significance than deeper level features and the ability to think about and communicate ‘ideas that matter.’
One exemplar was chosen because it showed serious limitations conceptually

22. The ‘concrete’ nature of exemplars very popular with students
“I think seeing it just makes you understand it more. Like, someone can stand there and say, 'You shouldn't do this and that' but until you've actually seen it then you don't know what that looks like.”
“I was just writing how I thought it should be done, just in my own head. But now I know what they are looking for and what I should be, how I should be writing, so yes, I think it's been helpful.”
“It helps, because you read it and think ‘Oh, they’ve put that, and that didn’t sound right, or that looks right’, and so it helps you listen out for your own writing.”

23. Interesting findings from our students’ responses in the workshop
Participant observation of phase 2
Many students couldn’t place in rank order
Survey of phase 3
Only 5 students in the sample (n=96) focused on the super-ordinate criterion in their feedback
“This misses the point and doesn’t answer the question”

24. GENERAL DISCUSSION POINTS

25. Do you use exemplars already, or could you envisage using them?
Generate a list of issues to bear in mind if you were planning to use exemplars in your own practice:
Design features: things to consider when choosing exemplars to show.
Process features: things to consider when putting exemplars into practice.

26. Design features: what would make a good exemplar
Design features: what would make a good exemplar? Some issues in the literature…
How to avoid overwhelming students with complexity? (Handley & Williams, 2011)
What’s the pedagogic point you’re trying to achieve? (Sambell, 2011)
How best to exemplify standards/avoid model answers/imitation? (Handley & Williams, 2011; Bell et al, 2013)
How to focus students’ attention on construction of knowledge in subject domain (deep-related features)not surface technical aspects? (Harrington et al, 2006)

27. Process features: putting exemplars into practice (c. f
Process features: putting exemplars into practice (c.f. Hendry et al,2012)
How do you create an interactive style?
How are students supported to mark and discuss a range of exemplars, with reasons?
How to open up staff perspectives and dialogue?
How much time to allow?
Are exemplars framed as a means or an end?

28. In sum....key principles of social constructivist approaches to the development of assessment literacy
actively make, in a scaffolded and supportive learning environment, evaluative judgments about the relative merit of concrete examples of student writing
participate in feedback interactions (Black & Wiliam, 2009) before they hand in their own writing for marking and grading
participate in assessment in a generative way (that is, providing, rather than simply receiving feedback) (Nicol, 2009)
‘Learn to see’ (Sadler, 2013), developing an understanding of the tacit knowledge and implicit rules of engagement that experts employ when assessing student work, which includes complex subject knowledge, situated in the context of the discipline (Orr, 2007; Bloxham, 2009).

29. References
Bell, A., Mladenovic, R & Price, M. (2013) Students’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars, Assessment & Evaluation in Higher Education, 38:7,
Bloxham, S. and West, A. (2007) ‘Learning to write in Higher Education: Students’ perceptions of an intervention in developing understanding of assessment criteria’, Teaching in Higher Education, 12: 1, 77–89.
Handley, K. & Williams, L. (2011): From copying to learning: using exemplars to engage students with assessment criteria and feedback, Assessment & Evaluation in Higher Education, 36:1,
Handley, K., Price, M. and Millar, J. (2008) Engaging Students with Assessment Feedback: Final Report for FDTL5 Project 144/03. Oxford Brookes University: Oxford.
Harrington, K. et al. (2006). ‘Using core assessment criteria to improve essay writing.’ In Bryan, C. and Clegg, K. (eds.), Innovative Assessment in Higher Education. London & New York: Routledge,
Hendry, G., Armstrong, S. & Bromberger, N. (2012): Implementing standards-based assessment effectively: incorporating discussion of exemplars into classroom teaching, Assessment & Evaluation in Higher Education, 37:2,
Hendry G and Anderson, J. (2013) Helping students understand the standards of work expected in an essay: using exemplars in mathematics pre-service
education classes, Assessment & Evaluation in Higher Education, 38:6,
Hendry., G., Bromberger, N and Armstrong, S (2011) Constructive guidance and feedback for learning: the usefulness of exemplars, marking sheets and different types of feedback in a first year law subject, Assessment & Evaluation in Higher Education, 36:1,1-11
Hendry (2013) in Merry et al,
Hounsell, D. (2007). ‘Towards more sustainable feedback to students.’ In: Boud, D. and Falchikov, N., eds. Rethinking Assessment in Higher Education. Learning for the Longer Term. London: Routledge,
Nicol, D. (2010) ‘From monologue to dialogue: Improving written feedback processes in mass
higher education’, Assessment and Evaluation in Higher Education, 35: 501–517.
Price, M., Handley, K., Miller, J and O’Donovan, B. (2011). Feedback: all that effort, but what is the effect? Assessment and Evaluation in Higher Education, 35 (3)
Price et al (20) Assessment Literacy, Oxford: ASKe.
Rust, C., O’Donovan, B. and Price, M. (2005) ‘A social constructivist assessment process model: How the research literature shows us this could be best practice’, Assessment and Evaluation in Higher Education, 30 (3): 231–40.
Sadler, D. (1989) Formative assessment and the design of instructional systems. Instructional Science 18 ,
Sambell, K. (2011) Rethinking feedback: An assessment for learning perspective, Bristol: ESCalate.
Sambell, K McDowell, L and Montgomery, C (2013) Assessment for Learning in Higher Education, Routledge.

30. Thanks for coming! Contact me for further information