26 March 2013 Welcome to: Transforming Assessment Pilot Scheme First Core Forum

Fire procedure. Cloakrooms. Refreshments. Catering. Administration and staffing. Wifi code. Car park code. Housekeeping

Dr Helen May, Academic Lead, HEA Dr Erica Morris, Academic Lead, HEA Professor Sally Brown, HEA Associate Professor Liz McDowell, HEA Associate Professor Margaret Price, HEA Associate Professor Chris Rust, HEA Associate Professor Brenda Smith, HEA Associate 3 Welcome and introductions

Stems from firm evidence that assessment is not fit for purpose. Aims to take a radical approach, recognising that it is time for significant reappraisal of assessment policy and practice. Builds on expertise, perspectives, and previous work. Takes an evidence informed approach. Encourages for assessment to be seen as an integral part of the learning experience. Background to A marked improvement

To further test and revise the review tool. To help institutions develop approaches to transforming assessment. To develop knowledge for the sector about the effectiveness of approaches to transform assessment. 5 Aims of the pilot scheme

66 Cyclical Phases of the Transforming Assessment Pilot Scheme Review PrioritiseDesign Implement & evaluate

Associate support and expertise from those who wrote A Marked Improvement. Expertise about institutional change. A series of interconnected one day core forums. A series of targeted one day forums. Tools and resources. To work with and learn from other institutions. To be a recognised leader in transforming assessment. 7 What does the scheme provide for participating institutions?

Provision of a task group to lead the design, implementation and evaluation of approaches to transform assessment. Engagement in all elements of the scheme, including all core forums, relevant targeted forums and visits. Provision of financial, action and evaluation plans and feedback following key events. Outputs: Recommendations for enhancing the tool. Worked examples and recommendations for applying the tenets into practice. Resources for those in learning and teaching roles. 8 What does the HEA expect from participating institutions?

For the whole working group. To promote inter-institutional exchange, collaboration and problem solving. To provide key information. Please ensure that the following dates are in your diary: 10 September March Core Forums

DateTenetLead 23 April 1 and 2 Helen May, Erica Morris, Sally Brown, Brenda Smith 8 May 3 and 4 Erica Morris, Chris Rust, Liz McDowell 14 May 5 and 6 Helen May, Sue Bloxham, Margaret Price. Targeted forums For two representatives from your working groups. Focus on particular tenets. Bringing together HEIs with same priority areas to facilitate joint reflection and development. Driven by your issues and concerns. Take place at the HEA offices in York.

Mutual respect, trust, support and encouragement Remain open minded and non-judgemental. Confidentiality Work within Chatham House Rule. Consent Obtain permission prior to disclosure of data, materials or other information. Rules of engagement

Aims and objectives of the day 12

To promote exchange, collaboration and problem solving between participating institutions. To introduce the expectations relating to the pilot scheme, supporters and participating institutions. To facilitate a collective understanding of the tenets. To enable the development of a vision for each institution, highlighting anticipated wider benefits in transforming assessment. To begin the process of creating action and evaluation plans, as an integral part of the scheme. 13 Objectives of the day

10:45Institutional introductions 11.25Panel: Unpacking the tenets Building a case for the tenets Lunch (and train spotting) 13.30Challenging the evidence 14.30Defining and communicating your approach Next steps 14 Programme

Will guide, question, listen, probe and advise (as required). Will understand your work and any issues or challenges you are facing on the scheme (more than others). Will speak up on your behalf, e.g. regarding content and design of the scheme. Will undertake a one day visit to your institution to help you progress. In addition, offers up to 1 day per institution of support over the course of the scheme, for you to instigate by or phone. 15 Supporter role

SupporterHEI(s) Sue BloxhamUniversity of Liverpool Sally Brown Teesside University Keele University Liz McDowellUniversity of Edinburgh Margaret PriceUniversity of Southampton Chris RustBucks New University Brenda SmithBirkbeck, University of London University of Roehampton 16 Supporters

Sally Brown and Liz McDowell Institutional introductions 17

Unpacking the tenets

Tenet 1: Assessment for learning Tenet 2: Ensuring assessment is fit for purpose Tenet 3: Recognising that assessment lacks precision Tenet 4: Constructing standards in communities Tenet 5: Integrating assessment literacy into course design Tenet 6: Ensuring professional judgements are reliable. Tenets

Helen May Building a case for the tenets 20

12345 Distribution of Adopters – Rogers, 1983 The members of each group have their own social and psychological characteristics that underlie their willingness to accept, adapt to, and implement change. Early adopters Innovators Early majority Late majority Laggards

Early Majority Late Majority Laggards Early Adopters Innovators The Chasm Number of Adopters Time 2.5%13.5%34% 16% Critical mass for widespread adoption is thought to be 15% to 20% Early Market Mainstream Market Late Market Adapted from Rogers (1983), Moore (1991), Geoghegan (1994)

Though people dont cross the chasm, we can try to reduce its size so that the rate of adoption doesnt stall too much as adoption moves from the early adopters to the early majority. The earlier we get the early majority on board, the narrower we have made the chasm.

How can we shrink the gap between the two key groups of people? Acknowledgement: Professor Alan Mortiboys; an educational and organisational development consultant

Changing hearts and minds…. Aiming to increase awareness Aiming to increase understanding Aiming to take action

Know your arguments for change….. What is unsatisfactory about the current situation? What is the evidence for change being required? What are you offering? What does it do? Know the arguments against change….. Who would be disinterested colleagues/ students? Why are they disinterested? Do they know they are disinterested? 26 Convincing others

Identify the anticipated outcomes or benefits for students of each of the following tenets: Table 1: Assessment for learning (Liz) Table 2: Ensuring assessment is fit for purpose (Chris) Table 3: Recognising that assessment lacks precision (Erica) Table 4: Constructing standards in communities (Margaret) Table 5: Integrating assessment literacy into course design (Brenda) Table 6: Ensuring professional judgements are reliable (Sally). Note: Please record your ideas on flip chart paper, noting any differences of interpretation or comments. Task: build a case to convince others

Transforming assessment in HE pilot scheme What comprises evidence? Brenda Smith, Sally Brown and Erica Morris 26 March 2013

How will we know the extent to which your engagement in the scheme has produced positive outcomes? What comprises good quality evidence of achievement through the scheme? How much evidence is enough? How can you assure the quality of your evidence? Key issues for discussion

Qualitative and quantitative evidence that provides convincing assurance that students have benefited from your engagement e.g.: improved NSS scores on assessment and feedback questions including positive free response comments; higher levels of satisfaction demonstrated through internal satisfaction surveys e.g. focus groups, module evaluations and feedback forums; positive comments from external examiners and Professional and Subject Body reviewers; demonstrably fewer student complaints and grievances in relation to assessment and feedback improved student retention, progression and achievement. 30 Positive outcomes may include:

Is your evaluation methodology sound and clearly thought through before commencing the scheme? Are the personnel and processes identified from the outset? Is your evidence verifiable and representative, (rather than relying on some positive comments from a few selected students)? Can you demonstrate triangulation of evidence, with benefits demonstrated from multiple sources (e.g. students across year groups or subjects, placement supervisors/ practice managers, employers, representatives of PSBs, validation teams, external examiners, QAA)? 31 How can you assure the quality of your evidence?

Reasons and Purposes (planning, managing, learning, developing, accountability); Uses (e.g. learning from good practice, staff development, strategic planning, PR, data management); Foci e.g. range of activities, aspects, emphasis, priority areas; Data and Evidence (numerical, qualitative, observational, case accounts); Audience for evaluation; Timing e.g. coincidence with decision making cycles, life cycle of projects Agency (Yourselves, external evaluators, combination). 32 RUFDATA: a planning framework for evaluation

Sufficient to: enable you within the timescale of your work to learn from and remediate any mistakes; convince your own HEI to scale up what has been achieved and ensure its sustainability; enable other HEIs to review your achievements, learn from them and emulate your successes; provide the HEA with confidence that they have invested well; But not so much that it is difficult to navigate and detracts by its volume from achievement of the scheme outcomes. 33 How much evidence is enough?

Pimmel1., R. Anderegg, B., Burkett, S., Watford, B., and Della-Pian, D. (2006) Evaluation of Educational Development Projects, ASEE/IEEE FIE2006 Conference The RUFDATA methodology x.php 34 References

Transformation of assessment – what does it look like? Margaret Price and Chris Rust26 March 2013

In your groups take it in turns to complete the following sentence: By the end of this scheme the most important thing we will have achieved will be ….. What I want to achieve?

You have 30 minutes to write outcomes on a flip chart. 1. By the end of this scheme the [noun*] will [verb]….. * Suggested list of nouns : Staff, students, culture, the institution specific programmes, departments, other 2. Repeat this for Within the next five years….. What do we want to achieve?

Look at your proposal again. As it is currently written will what you are proposing to do achieve the outcomes you have just identified. Is the focus on the right tenets(s) to achieve your outcomes? How will a focus on your chosen tenet(s) support transformation of assessment? Will the initial focus provide a good foundation to address broader and long term transformation? Identify changes you might need to make to your plans. Will our plans achieve it?

Continue these conversations to reach a common view that you can communicate within your institution. 39 What next?

Next Steps Helen May and Erica Morris 40

Let us have your reflections and ideas, through the feedback survey, open until 19 th April Confirm which tenet(s) you will focus on and who will be representing your task group at the 1 st targeted forum by 8 th April. On or before 30 th April, please also provide us with an action plan, financial breakdown and your revised web profile. Address any admin/technical queries about the pilot scheme to Contact your supporter with any specific queries. Draw on one anothers expertise and experience. Next Steps 41

Your action plan should contain the following key elements: Areas for focused development (what). Anticipated outcomes (why). Approaches and activities for achieving the scheme outputs and implementing transformative change (how). Relevant milestones (when). Evaluation (so what). Plans for dissemination and exit/sustainability (after). Action plan

Your financial statement should contain the following: Breakdown of activity directly related to transforming assessment and participating in the scheme. Estimated or actual costs. Dates of spend. Please note – funds should not be used for work external to the institution (eg dissemination). 43 Financial breakdown

This is your external communication about your work to the sector. Please revise as a result of the day. Provide a balance between: Your institutional context. Your priority areas for development. Rationale for participation. Word limit: 250 words. Incorporate your statement defining and communicating your approach. 44 Web profiles

By 31 st May: Recommendations as to how the review tool can be enhanced. For HE providers (not HEA), length 1-2,000 words This will need to be reflective on three or more of the following: Process (e.g. usability, approach) Clarity (e.g. terminology, underpinning theories, accessibility) Structure (e.g. target audience, part A and B) Scope (e.g. coverage, missing elements) Aligned agendas (eg academic integrity, inclusion, employability, internationalisation and students as partners). 45 Looking ahead to the first deliverable….

Any questions or issues about the scheme? 46