2. SQAs Approach to Quality Assurance of Assessment Matthew McCullagh Quality Manager Welcome

3. Why are we here? To introduce you to SQAs New Approach to Quality Assurance To help you understand the key changes to Quality Assurance process To help you become familiar with: –The new Quality Assurance Criteria –The new Confidence Statements –How visits are planned, conducted and reported

4. Monday 10 th September 2012 9.00 Coffee and Registration 9.40 Introduction 9.45New Approach to QA Overview, Aims, Objectives and Overview of Event 10.30Break 10.45 Systems Verification & Evidence 12.00 Lunch 1.30 Systems Verification & Evidence 2.45 Break 3.00 Qualification Verification & Evidence 4.15 Summary & Outline of Day 2 4.30Close

5. SQAs Approach to QA SQA is moving to introducing a new approach to Quality Assurance of qualifications delivered in China It is based on SQAs Principles of Quality Assurance: Open, fair and transparent Aimed at risk reduction Proportionate Intelligence led A shared responsibility Constantly reviewed and improved

6. What is the new approach to Quality Assurance SQA now operates a transparent, intelligence-led, risk based model for quality assurance of approved centres and potential centres delivering our HND qualifications in China.

7. Timeline of Implementation of New Approach in China Overview to Centre staff in Beijing and Xiamen Launch of new approach Overview to EVs at central verification events Specialist training for Centre staff and EVs Implementation new approach to QA September 2011 May 2012 September 2012 September 2012-2013 March 2012

8. New Quality Criteria Five Categories: –Management of a centre –Resources –Candidate Support –Assessment & Verification –Records/Data Management All signposted to four QA processes All pre-rated as High/Medium/Low impact All supported by possible sources of evidence

9. The New Approach to Quality Assurance: The Four Processes Systems Approval Qualification Approval Systems Verification Qualification Verification

10. High/Medium/Low Impact Criteria All criteria are important However, some are more important than others Those that are most important are rated as High Impact criteria Those that are less important are rated as Low Impact criteria In between these two are medium impact criteria. Failure to meet a High Impact criteria will have a greater impact on the outcome of the visit

11. Sufficiency of evidence No Systems Verifier or Qualification Verifier can change the impact rating of a criteria. For each criterion you verify, you must decide whether the centre has presented you with: –Sufficient evidence to meet the criterion –Some, but insufficient evidence to meet the criterion –No evidence to meet the criterion

12. Good Practice and Guidance As part of the verification visit, you should identify good practice that is evident in the centre. You should also give advice and guidance to help centre staff improve the delivery of our qualifications This is a mandatory part of your role as a Qualification Verifier

13. Sufficiency of evidence We will support you and help you to understand what sort of evidence centres can provide to meet each criterion Once you have made your decision regarding the sufficiency of evidence, SQA staff will then calculate our level of confidence This will depend on the impact level of the criteria and the sufficiency of the evidence presented

14. Introduction of Confidence Statements In relation to Qualification Verification Confidence Statement In relation to Systems Verification High level of confidence in the maintenance of SQA standards within this Verification Group High Level of Confidence High level of confidence in the systems that support the maintenance of SQA standards within this centre Broad confidence in the maintenance of SQA standards within this Verification Group Broad Confidence Broad confidence in the systems that support the maintenance of SQA standards within this centre Reasonable confidence in the maintenance of SQA standards within this Verification Group, although moderate risks exist within the following categories: (insert appropriate category) Reasonable Confidence Reasonable confidence in the systems that support the maintenance of SQA standards within this centre, although moderate risks exist within the following categories: (insert appropriate category) Minimal confidence in the maintenance of SQA standards within this Verification Group as significant risks exist within the following categories: (insert appropriate category) Minimal Confidence Minimal confidence in the systems that support the maintenance of SQA standards within this centre as significant risks exist within the following categories: (insert appropriate category) No confidence in the maintenance of SQA standards within this Verification Group as severe risks exist within the following categories: (insert appropriate category) No Confidence No confidence in the systems that support the maintenance of SQA standards within this centre as severe risks exist within the following categories: (insert appropriate category)

15. How Confidence Statements are calculated Confidence statements are given for the outcome of each Systems Verification and Qualification Verification visit for each of the categories of criteria verified These are based on the sufficiency of the evidence provided to the you A summary statement of confidence is also given, based on the confidence statements for each category

16. What if non-compliances are identified? Where there is No evidence or insufficient evidence to meet a criterion, you must agree with centre staff, during the visit: –the action the centre needs to take in order to become compliant. –The evidence they need to produce –Where to send this evidence –The date by which the action must be taken

17. What happens then? SQA will ask you to consider the evidence the centre sends us and ask you to consider whether it is Sufficient or insufficient. Depending on the sufficiency of the evidence, the Confidence Statement will be re-calculated.

18. Sanctions If the outcome of a verification visit results in a Confidence Statement of Reasonable, Minimal or No Confidence, SQA staff may decide to place a sanction on a centre until our confidence increases to Broad or High. Our confidence may decrease if a centre fails to meet action points agreed during a verification visit. Sanctions will only be used where necessary.

19. Sanctions In relation to Qualification Verification Confidence Statement In relation to Systems Verification High Level of Confidence Entry in Action Plan Broad Confidence Entry in Action Plan Suspension of specific qualification certification: by verification group by qualification Suspension of approval application by verification group Suspension of direct certification claim status: by verification group by qualification Reasonable Confidence Suspension of all existing qualification approval Suspension of centre certification Suspension of certification for all qualifications with assessed components Suspension of qualification approval application - whole centre Suspension of specific qualification approval: by verification group by qualification Minimal Confidence Withdrawal of all existing specific qualification approval Withdrawal of specific qualification approval: by verification group by qualification No Confidence Withdrawal of centre approval

20. Benefits of new approach for centres Open and transparent to all Clear guidance available to verifiers Proportionate response where issues are identified Focused on Good Practice and improvement as well as compliance

21. Questions

22. Introducing the Quality Assurance Criteria Systems Verification Criteria You have been provided with criteria from the two categories of the Systems Verification Criteria (Management of a Centre and Resources). In addition to this you have been provided with many examples of evidence. Working in your groups please match the appropriate examples of evidence to the relevant criteria. Formulate a list of points on flip chart paper to help you explain the chosen evidence for each criterion to your colleague verifiers. The summary should also include possible sources of evidence. Timings: 30 minutes discussion and research and flip –charting your points 30 minutes to feedback Please appoint a spokesperson(s) to present the flip-charted points. Each group will be assigned a facilitator to answer any questions/queries you may have.

23. New Approach to Quality Assurance Welcome Back

24. Introducing the Quality Assurance Criteria Systems Verification Criteria You have been provided with criteria from the two categories of the Systems Verification Criteria (Management of a Centre and Resources). In addition to this you have been provided with many examples of evidence. Working in your groups please match the appropriate examples of evidence to the relevant criteria. Formulate a list of points on flip chart paper to help you explain the chosen evidence for each criterion to your colleague verifiers. The summary should also include possible sources of evidence. Timings: 30 minutes discussion and research and flip –charting your points 30 minutes to feedback Please appoint a spokesperson(s) to present the flip-charted points. Each group will be assigned a facilitator to answer any questions/queries you may have.

25. Introducing the Quality Assurance Criteria Qualification Verification Criteria You have been provided with criteria from the two categories of the Systems Verification Criteria (Management of a Centre and Resources). In addition to this you have been provided with many examples of evidence. Working in your groups please match the appropriate examples of evidence to the relevant criteria. Formulate a list of points on flip chart paper to help you explain the chosen evidence for each criterion to your colleague verifiers. The summary should also include possible sources of evidence. Timings: 30 minutes discussion and research and flip –charting your points 30 minutes to feedback Please appoint a spokesperson(s) to present the flip-charted points. Each group will be assigned a facilitator to answer any questions/queries you may have.

26. Tuesday 11 th September 2012 9.30Welcome 9.35Process: Preparing & Planning a Visit 10.50Break 11.00Process: Conducting a Visit 12.20Lunch 1.50Decision Making/Action Planning/Reporting 3.05Break 3.20Question & Answer 4.20Close

27. Questions

28. Planning for systems/qualification Verification The systems/qualification verifier role activities Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

29. Verification planning – the centres involvement Each centre plays a vital role at the planning stages of a visit. They help ensure An optimum time is selected for visits to take place There is enough evidence for the verifier to sample (for the verifier and centre to gain value from the visit)

30. Visit Planning - Qualification Verification SQA (China Office) agrees optimum date (s) for visiting verification. This is an agreement between SQA, each centre and each QV SQA selects Units to be verified and communicates this to each centre (copying to SQA China Office) Qualification Verifier contacts centre to confirm visit date(s) and travel arrangements SQA (China Office) confirms to each QV, the centre(s) they should visit, the date of each visit, and the Units they should verify

31. Visit Planning - Systems Verification Accept centre allocation from SQA Make initial contact with each centre Formulate Visit Plan Request information from centre Send visit planning documentation

32. Workshop 4: Identifying the preparations a centre would make. Before the verification visit. In your groups, identify the types of preparations a centre would usually make prior to receiving a verification visit. Write down the points you have identified on the post-its (one point per post-it) provided. A facilitator will collect post-its from all of the groups to be used to support a plenary discussion. In formulating your points, you may wish to refer to SQAs New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points) 15 minutes plenary discussion Each group will be assigned a facilitator to answer any questions/queries you may have.

33. Visit Planning – Qualification Verification: Applying a sampling frame SQA is responsible for systematically sampling on a per centre basis: Selected Units (against selected cohorts) Assessment/internal verification decisions and practice Methods/Instruments of assessment Problematic/revised units Locations where assessment takes place Evidence of candidates work for the Units selected Documented evidence of policies/procedures (implementation)

34. Carry out systems/qualification verification The systems/qualification verifier role activities Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

35. The role of the systems/qualification verifier in carrying out the visit Agreeing the agenda and running order Conducting sampling activities Judging the sufficiency of evidence

36. Workshop 5: Identifying a centres involvement during a verification visit During the verification visit In your groups, using the grid provided, identify how centres might get involved/support each of the following stages when the verifier is conducting a verification visit. In formulating your points, you may wish to refer to SQAs New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points) 15 minutes plenary discussion Each group will be assigned a facilitator to answer any questions/queries you may have.

37. Agreeing the agenda and running order The centres role: Discussing agenda/timings/order Agreeing level of involvement when verifier is sampling of evidence Being clear on arrangements for sampling/interviews Communicating arrangements to relevant centre staff

38. Conducting sampling activities The centres role: Providing access to all required documentation (policies, procedures, candidate evidence, assessment records etc) Making suitable accommodation available for sampling and interviews Allowing the right personnel to be available at the relevant times Maintaining an absence if or when presence is not required

39. In new NAQA Guidance - Conducting sampling activities Complete Units (all Outcomes complete) Incomplete Units (some Outcomes complete) Evidence not yet internally verified Assessment/verification decisions Assessment and verification practice Interviews with assessors/verifiers/candidates Referencing of evidence to standards

40. Judging evidence in relation to each criterion Centre provides (through documentation and or interviews) as much relevant information as possible in support of each criterion.

41. Access Increased understanding More questions Access to information

42. Carry out systems/qualification verification The systems/qualification verifier role activities Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

43. Communicating verification decisions The verifiers roles involves: Delivering balanced feedback Explaining their verification decisions

44. Communicating verification decisions Balanced feedback – the verifiers aim is to provide an objective reflection of each centres level of compliance in relation to each criterion.

45. Communicating verification decisions Specifically, feedback will cover: criteria that was met good practice recommendations agreed action points

46. Communicating verification decisions SQA consider the explanation of verification decisions to be an important part of the feedback process. Centres should be clear as to how decisions have been made in relation to each criterion.

47. Clarifying verification decisions The use of questioning is key. If questions are carefully worded, they can benefit both parties. Some examples: Can you explain this point again, I am not sure if I have fully understood it? Did I mention that we also have evidence in file ……….? Therefore, we met this criterion because …(paraphrase)? Can you give me a description of the type of evidence you are looking for, so that I can fully meet this action point? We proposed that this evidence…. covered the unit specification, was our thinking along the right lines?

48. Agreeing action points Specific: Centres should be clear in terms of the action they need to take to close-off the action point. Measurable: How will the centre know that the action point has been met, what will be the measure of success? Achievable: The centre must have sufficient time in which to achieve the agreed action (consider who is involved, the amount of work required). Relevant: The action must directly relate to the criterion. Time bound: Agreement of specific and realistic date(s) for the action to be completed by.

49. Workshop 6:After the verification visit After the verification visit Consider the activities a centre would undertake as a result of the outcomes of a verification visit In particular consider : what should happen who should be responsible when it should happen how you know activities have been completed successfully Write your points on the post-its provided (one point per post-it). In formulating your points, you may wish to refer to SQAs New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (recording your points) 15 minutes plenary discussion Each group will be assigned a facilitator to answer any questions/queries you may have.

50. After the verification visit: centre responsibilities Once verification has taken place, the centre is responsible for following up on the outcomes of the verification report. Those responsibilities would include: Dissemination of the report Allocating responsibility for actions Tracking progress and achievement of actions

51. SQAs Approach to Quality Assurance of Assessment Welcome Back

52. QIIPS: Quality, Integrity, Innovation, Partnership & Service Sounds like: QUIPS – a dictionary definition: A clever remark often prompted by the occasion.

53. SQA Values Quality Integrity Innovation Partnership Service

54. SQA Values Quality As custodians of the standards we verify, we continually strive to maintain consistent quality in their assessment.

55. SQA Values Integrity We use open and honest communication with internal and external customers (centres), promoting transparency, and building trust with others. We take a standardised approach to verification as laid down by SQA and agreed by consensus within our Qualification Verification team, even if our personal opinions may at times conflict.

56. SQA Values Innovation We view unfamiliar approaches to assessment and verification with an open mind, whilst ensuring quality is maintained. We believe that creativity should be seen not as a threat to quality assurance but as an opportunity for improving upon existing practice.

57. SQA Values Partnership We understand that by working in partnership with SQA centres, we will achieve common goals of excellence and consistency in assessment.

58. SQA Values Service We maintain a professional approach with SQA centres at all times, regardless of the challenges we might face in carrying out our work.

59. Workshop 7 Values Exercise

60. Summary To introduce you to SQAs New Approach to Quality Assurance To help you understand the key changes to Quality Assurance process To help you become familiar with: –The new Quality Assurance Criteria –The new Confidence Statements –How visits are planned, conducted and reported

61. Questions