1. Assessment and Feedback Developing Graduate Attributes Professor David Nicol Facilitator: QAA Scotland: Assessment and Feedback Deputy Director Centre for Academic Practice and Learning Enhancement (CAPLE) University of Strathclyde d.j.nicol@strath.ac.uk University of Strathclyde,28 th September 2010 Queen Margaret University, 29 th September 2010

2. Complex Appraisal and Graduate Attributes 1. Graduates for the 21 st century 2. Implementation in the disciplines – creating the right ‘decision space’ 3. Institutional barriers and opportunities

3. Graduates for the 21 st century Graduate attributes descriptors The Australian influence (Sydney: Curtin: Melbourne) Scottish HE developments Questions 1. What underpins attribute development? 2. How would you monitor and evaluate attribute development? How does all this relate to Royce Sadler’s ideas?

4. The Sydney attributes

5. The Sydney Attributes 1. Research and Inquiry graduates will be able to create new knowledge and understanding through the process of research and inquiry 2. Information Literacy graduates will be able to use information effectively in a range of contexts 3. Personal and Intellectual Autonomy graduates will be able to work independently and sustainably, in a way that is informed by openness, curiosity and a desire to meet new challenges. 4. Ethical, Social and Professional Understanding graduates hold personal values and beliefs consistent with their roles as members of local, national and international and professional communities. 5. Communication graduates will use and value communication as a tool for negotiation and creating new understanding, interacting with others, and furthering their own learning.

6. The argument ‘…..the core process underpinning all the… [Sydney].. attributes is the students’ ability to think for themselves, to be self-reliant and able to evaluate and make judgements about their own thinking and actions: that is, all the attributes require that students are self- regulating learners’ [Nicol, 2010, p4]. Indeed, these ideas are already contained under the attributes descriptors at Sydney. Reference: Nicol, D (2010) The foundation for graduate attributes: developing self- regulation through self and peer processes, Published by QAA for HE. http://www.enhancementthemes.ac.uk/documents/G21C/Assessment_150910.pdf

7. The Sydney Attribute Descriptors Research and Inquiry Be able to identify, define and analyse problems and create processes to solve them Be able to exercise critical judgment and critical thinking in creating new understanding. Be creative and imaginative thinkers Have an informed respect for the principles, methods, standards, values and boundaries of their discipline and the capacity to question these Be able to critically evaluate existing understandings and recognise the limitations of their own knowledge

8. The Sydney Attribute Descriptors Information literacy Recognise the extent of information needed Locate needed information efficiently and effectively Evaluate information and its sources Use information in critical thinking and problem solving contexts to construct knowledge Understand economic, legal, social and cultural issues in the use of information Use contemporary media and technology to access and manage information

9. The Sydney Attribute Descriptors Personal and Intellectual Autonomy Be intellectually curious and able to sustain intellectual interest Be capable of rigorous and independent thinking Be open to new ideas, methods and ways of thinking Be able to respond effectively to unfamiliar problems in unfamiliar contexts Be able to identify processes and strategies to learn and meet new challenges Be independent learners who take responsibility for their own learning, and are committed to continuous reflection, self-evaluation and self-improvement Have a personal vision and goals and be able to work towards these in a sustainable way

10. The Sydney Attribute Descriptors Ethical, Social and Professional Understanding Strive for truth, honesty, integrity, open-mindedness, fairness and generosity Acknowledge their personal responsibility for their own value judgments and behaviour Understand and accept social, cultural, global and environmental responsibilities Be committed to social justice and principles of sustainability Have an appreciation of and respect for diversity Hold a perspective that acknowledges local, national and international concerns Work with, manage, and lead others in ways that value their diversity and equality and that facilitate their contribution to the organisation and the wider community

11. The Sydney Attribute Descriptors Communication Use oral, written, and visual communication to further their own learning Make effective use of oral, written and visual means to critique, negotiate, create and communicate understanding Use communication as a tool for interacting and relating to others

12. Developing graduate attributes Requires that students have regular opportunities to: 1. critically evaluate the quality and impact of their own work, both during and after its production (e.g. academic texts, problem solutions, designs) 2. critically evaluate the quality and impact of the work of others (e.g peers). This is an enabling condition for 1 and is a valued skill sought by employers.

13. Feedback still has a role to play Generating own feedback and giving feedback to others is more productive than receiving it It is cognitively more demanding: cannot do this passively Puts students in role of teacher Students actively exercise criteria from many viewpoints Peers see a wide sample of work (alternative approaches, remedies) That recognise that quality is not a fixed attribute – a range of possibilities

14. Implementing complex appraisal in HE 1. Make complex appraisal (self/peer review) explicit 2. Consider a whole-institution approach 3. Make implementation easy – many ways to do it, many pitfalls and its management can be time consuming 4. Bring the students on-board – clarify their role as feedback producers 5. Address the NSS at the sector level 6. Collaborate and share expertise across Scottish HE

15. 1. Make core processes explicit Course learning outcomes rarely state: At the end of this module: …you will be able to evaluate critically the quality and/or impact of your own work …you will be able to evaluate critically the quality and impact of the work produced by others

16. Benefits of this approach to attributes development 1. it identifies a core focus for curriculum design and an efficient solution to attributes implementation 2. addresses co-curricular activities where students must evaluate their own performance without the help of the teacher 3. supports transfer of attributes and skills to new contexts: this depends on self-monitoring and self-assessment 4. it aligns attributes with most university policies which emphasise autonomy and independence 5. makes mapping of attributes easier – specific attributes brought into play by fine fine-tuning of task parameters/learning activities (e.g. ethical awareness, internationalisation)

17. 2. Develop an institutional approach Texas A&M University (TAMU) see peer review as not only developing higher order thinking but also student writing skills At TAMU, over the last 3 years, over 15,300 students have engaged in peer review in 340 courses led by 368 instructors TAMU use Calibrated Peer Review software developed by UCLA (this is widely used across US) TAMU provide a dedicated support service to help staff use the software and introduce it to students

18. MAJORS USING CPR at TAMU TM Accounting Ag. Economics Animal Science Anthropology Archeology Biochemistry Biology Botany Bus. Admin. Ctr. Acad. Enh. Chemistry English Ed. Curriculum Ed. Psychology Engineering Film French German Kinesiology Learn. Comm. Math Microbiology Nutrition Physics Poultry Science Psychology Reading Secondary education Vet Integr. Bio Sci. Wildlife and Fish Sci. Zoology This slide taken from a presentation by Dr. Wendy Keeney-Kennicutt, Calibrated Peer Review Administrator, TAMU

19. 3. Make self/peer review easy to implement Many ways to implement peer review – disciplinary focus and educational process (e.g. rubrics, calibration, responsiveness, self-review etc.) Raises many issues (grading and development: collaboration versus collusion) Time consuming to implement when numbers are large

20. Peer Review in Education Evaluation [PEER] PEER project is funded by JISC (July 2010) and led by University of Strathclyde. The aims are to: review evidence base for peer review develop educational designs for peer review (and self-review) identify software support for peer review pilot implementations of large-scale peer review produce guidelines for the HE/FE – why do it, how to do it, pitfalls, and software possibilities See http://www.jisc.ac.uk/whatwedo/programmes/elearning/ltig/pe er.aspx http://www.jisc.ac.uk/whatwedo/programmes/elearning/ltig/pe er.aspx

21. 4. Bring the students on-board as feedback producers

22. 4. Bring the students on-board

23. 5. Address the NSS A distorting influence….! Investigate what the NSS is measuring Consider developing and piloting a Scottish National Student Survey alongside the NSS

24. Share ideas across the Scottish HE sector This can begin today …. Share ideas across the Scottish HE sector

25. Some of my Publications Nicol, D (2010) From monologue to dialogue written feedback in mass higher education. Assessment and Evaluation in Higher Education. 35(5), 501 -517 Nicol, D and Draper, S (2010), A blueprint for transformational organisational change in HE: REAP as a case study (see reap.ac.uk website) Nicol, D (2009), Transforming assessment and feedback: Enhancing integration and empowerment in the first year, Published by Quality Assurance Agency, Scotland (http://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_Transform ingAssess.pdfhttp://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_Transform ingAssess.pdf Nicol, D (2009), Assessment for learner self-regulation: Enhancing achievement in the first year using learning technologies, Assessment and Evaluation in Higher Education, 34(3), 335-352 Nicol, D (2007), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology, 38(4), 668-678 Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.31(1), 53-64. Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge. Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218. See also www.reap.ac.uk for copies.www.reap.ac.uk