2. SQA’s Approach to Quality Assurance of Assessment
Matthew McCullagh
Quality Manager
Welcome 欢迎

3. Why are we here?
To introduce you to SQA’s New Approach to Quality Assurance
To help you understand the key changes to Quality Assurance process
To help you become familiar with:
The new Quality Assurance Criteria
The new Confidence Statements
How visits are planned, conducted and reported

4. Thursday 6th September 2012 9.00 Coffee and Registration
Introduction, New Approach to QA Overview, Aims and Objectives of Training Days
Preparation for Qualification Verification Activity, Attitudes and Behaviours, KPMs
Break
Preparation for Qualification Verification Activity,
Attitudes and Behaviours
Planning for Qualification Verification Activity – the theory
Lunch
Conducting the Qualification Verification Visit – the theory
Break
Introducing the Quality Assurance Criteria
Quality Assurance Criteria Workshop
Close

5. Friday 7th September 2012
9.00 Coffee and Registration 9.30 Feedback: The Quality Assurance Criteria Break Decision Making Lunch 1.30 Communicating Decisions 2.00 Theory of Completing Excel Reports/Action Plans 2.30 Break 2.45 Writing effective Verifier Reports 3.45 Case Study Assignment 4.00 Summary and Q & A 4.30 Close

6. SQA’s Approach to QA
SQA is moving to introducing a new approach to Quality Assurance of qualifications delivered in China
It is based on SQA’s
Principles of Quality Assurance:
Open, fair and transparent
Aimed at risk reduction
Proportionate
Intelligence led
A shared responsibility
Constantly reviewed and improved

7. What is the new approach to Quality Assurance
SQA now operates a transparent, intelligence-led, risk based model for quality assurance of approved centres and potential centres delivering our HND qualifications in China.

8. Timeline of Implementation of New Approach in China
Overview to Centre staff in Beijing and Xiamen
Launch of new approach
Overview to EVs at central verification events
Specialist training for Centre staff and EVs
Implementation new approach to QA
September 2011
March
2012
May
2012
September 2012
September

9. New Quality Criteria Five Categories:
Management of a centre
Resources
Candidate Support
Assessment & Verification
Records/Data Management
All signposted to four QA processes
All pre-rated as High/Medium/Low impact
All supported by possible sources of evidence

10. The New Approach to Quality Assurance: The Four Processes
Systems Approval
Qualification Approval
Systems Verification
Qualification Verification

11. High/Medium/Low Impact Criteria
All criteria are important
However, some are more important than others
Those that are most important are rated as High Impact criteria
Those that are less important are rated as Low Impact criteria
In between these two are medium impact criteria.
Failure to meet a High Impact criteria will have a greater impact on the outcome of the visit

12. Sufficiency of evidence
No Systems Verifier or Qualification Verifier can change the impact rating of a criteria.
For each criterion you verify, you must decide whether the centre has presented you with:
Sufficient evidence to meet the criterion
Some, but insufficient evidence to meet the criterion
No evidence to meet the criterion

13. Good Practice and Guidance
As part of the verification visit, you should identify good practice that is evident in the centre.
You should also give advice and guidance to help centre staff improve the delivery of our qualifications
This is a mandatory part of your role as a Qualification Verifier

14. Sufficiency of evidence
We will support you and help you to understand what sort of evidence centres can provide to meet each criterion
Once you have made your decision regarding the sufficiency of evidence, SQA staff will then calculate our level of confidence
This will depend on the impact level of the criteria and the sufficiency of the evidence presented

15. Introduction of Confidence Statements
In relation to Qualification Verification
Confidence Statement
In relation to Systems Verification
High level of confidence in the maintenance of SQA standards within this Verification Group
High Level of Confidence
High level of confidence in the systems that support the maintenance of SQA standards within this centre
Broad confidence in the maintenance of SQA standards within this Verification Group
Broad Confidence
Broad confidence in the systems that support the maintenance of SQA standards within this centre
Reasonable confidence in the maintenance of SQA standards within this Verification Group, although moderate risks exist within the following categories:
(insert appropriate category)
Reasonable Confidence
Reasonable confidence in the systems that support the maintenance of SQA standards within this centre, although moderate risks exist within the following categories:
Minimal confidence in the maintenance of SQA standards within this Verification Group as significant risks exist within the following categories:
Minimal Confidence
Minimal confidence in the systems that support the maintenance of SQA standards within this centre as significant risks exist within the following categories:
No confidence in the maintenance of SQA standards within this Verification Group as severe risks exist within the following categories:
No Confidence
No confidence in the systems that support the maintenance of SQA standards within this centre as severe risks exist within the following categories:

16. How Confidence Statements are calculated
Confidence statements are given for the outcome of each Systems Verification and Qualification Verification visit for each of the categories of criteria verified
These are based on the sufficiency of the evidence provided to the you
A summary statement of confidence is also given, based on the confidence statements for each category

17. What if non-compliances are identified?
Where there is No evidence or insufficient evidence to meet a criterion, you must agree with centre staff, during the visit:
the action the centre needs to take in order to become compliant.
The evidence they need to produce
Where to send this evidence
The date by which the action must be taken

18. What happens then?
SQA will ask you to consider the evidence the centre sends us and ask you to consider whether it is Sufficient or insufficient.
Depending on the sufficiency of the evidence, the Confidence Statement will be re-calculated.

19. Sanctions
If the outcome of a verification visit results in a Confidence Statement of Reasonable, Minimal or No Confidence, SQA staff may decide to place a sanction on a centre until our confidence increases to Broad or High.
Our confidence may decrease if a centre fails to meet action points agreed during a verification visit.
Sanctions will only be used where necessary.

20. Sanctions In relation to Qualification Verification
Confidence Statement
In relation to Systems Verification
High Level of Confidence
Entry in Action Plan
Broad Confidence
Suspension of specific qualification certification:
by verification group
by qualification
Suspension of approval application by verification group
Suspension of direct certification claim status:
Reasonable Confidence
Suspension of all existing qualification approval
Suspension of centre certification
Suspension of certification for all qualifications with assessed components
Suspension of qualification approval application - whole centre
Suspension of specific qualification approval:
Minimal Confidence
Withdrawal of all existing specific qualification approval
Withdrawal of specific qualification approval:
No Confidence
Withdrawal of centre approval

21. Benefits of new approach for centres
Open and transparent to all
Clear guidance available to verifiers
Proportionate response where issues are identified
Focused on Good Practice and improvement as well as compliance

22. Questions

23. SQA’s Approach to Quality Assurance of Assessment
David Pirnie
Lead Verifier
Welcome 欢迎

24. Qualification Verification
The Qualification Verifier role activities
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

25. Preparing for Qualification Verification
The Qualification Verifier role activities
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

26. Being prepared “The best preparation for good work
tomorrow is to do good work today”.
Elbert Hubbard

27. Preparing for Qualification Verification
Before carrying out any activities, verifiers are
required to have a full understanding of their role
and responsibilities. This involves acquiring
knowledge and keeping it up to date.

28. Preparing for Qualification Verification
Required knowledge – three key areas:
Knowledge of your subject area
Centre roles and centre responsibilities
Qualification Verification

29. Preparing for Qualification Verification
Knowledge of your subject area:
Have a common interpretation of the Unit Standards including the Evidence Requirements for each unit you have been assigned to verify
Know how to access Arrangements Documents, Instruments of Assessment and Marking Schemes to gain a full understanding of content

30. Preparing for Qualification Verification
Knowledge of centres roles and responsibilities:
Centre Co-ordinators
Assessors
Internal verifiers

31. Preparing for Qualification Verification
Knowledge of the Qualification Verifier role:
Role Profile
Sources of support
Verification policies and procedures
Values

32. QIIPS: Quality, Integrity, Innovation, Partnership & Service
Sounds like:
“QUIPS” – a dictionary definition:
“A clever remark often prompted by the occasion”.

33. SQA Values
Quality
Integrity
Innovation
Partnership
Service

34. SQA Values Quality As custodians of the standards we verify, we
continually strive to maintain consistent quality in
their assessment.

35. SQA Values
Integrity
We use open and honest communication with internal and external customers (centres), promoting transparency, and building trust with others.
We take a standardised approach to verification as laid down by SQA and agreed by consensus within our Qualification Verification team, even if our personal opinions may at times conflict.

36. SQA Values
Innovation
We view unfamiliar approaches to assessment and verification with an open mind, whilst ensuring quality is maintained.
We believe that creativity should be seen not as a threat to quality assurance but as an opportunity for improving upon existing practice.

37. SQA Values Partnership
We understand that by working in partnership with SQA centres, we will achieve common goals of excellence and consistency in assessment.

38. SQA Values
Service
We maintain a professional approach with SQA centres at all times, regardless of the challenges we might face in carrying out our work.

39. Preparing for Qualification Verification
More information on the preparation phase of
Qualification Verification can be found in the first
section of SQA’s New Approach to Quality
Assurance: Guidance on Visiting Verification For
HN Qualifications in China

40. Workshop
Values Exercise

41. Planning for Qualification Verification
The Qualification Verifier role activities
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

42. “Failing to plan is planning to fail”.
Alan Lakein

43. Visit Planning: Sampling
SQA cannot always verify all HN Units, for all
candidates, on every visit to a centre, therefore a
sampling strategy has to be adopted.
Purpose of sampling:
To establish that there is a common interpretation of the standards within each centre and that the assessment and verification systems allow valid, reliable and fair assessment decisions to be made.

44. Visit Planning: Applying a sampling frame
SQA is responsible for systematically sampling on
a percentre basis:
Selected Units
Assessment decisions and practice
Methods/Instruments of assessment
Problematic/revised units
Locations where assessment takes place
Evidence of candidates across Units
Documented evidence of policies/procedures (implementation)

45. Visit Planning
SQA (China Office) agrees optimum date (s) for visiting verification. This is an agreement between SQA, each centre and each QV
SQA selects Units to be verified and communicates this to each centre (copying to SQA China Office)
SQA (China Office) confirms to each QV,
the centre(s) they should visit, the date of each visit, and the Units they should verify
Qualification Verifier contacts centre to confirm visit date(s) and travel arrangements

46. New Approach to Quality Assurance
Welcome Back
欢迎回来

47. Carry out Qualification Verification
The Qualification Verifier role activities
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

48. Carry out Qualification Verification
Discussion point
Consider the verification activities you perform
from the time of arrival until the end of the visit.
Contribute to the discussion by letting us know
what kind of techniques/approaches you have
used that you have found particularly beneficial.

49. Carry out the visit Prepare yourself for the visit
Agree agenda and running order of visit
Conduct sampling activities
Judge the sufficiency of evidence

50. Conduct sampling activities
Complete Units (all Outcomes complete)
Incomplete Units (some Outcomes complete)
Evidence not yet internally verified
Assessment/verification decisions
Assessment and verification practice
Interviews with assessors/verifiers/candidates
Referencing of evidence to standards

51. Introducing the Quality Criteria
Introducing the New Approach to Quality
Assurance Criteria
Introducing the Qualification Verification Guidance

52. Introducing the Quality Criteria
In your groups, discuss and familiarise yourselves with the criteria you have been allocated.
Formulate a list of summary points on flip chart paper to help you explain the purpose of each criterion to your colleague verifiers. The summary should also include possible sources of evidence.
In formulating your points, you are asked to use SQA’s New Approach to Quality Assurance: Guidance on Visiting Verification for HND Qualifications in China as your information source.
Timings:
30 minutes discussion and research
20 minutes flip-charting your points
Please appoint a spokesperson(s) to present the flip-charted points which will be presented in the first session tomorrow morning.
Each group will be assigned a facilitator to answer any questions/queries you may have.

53. Questions

54. The New Approach to Quality Assurance
Welcome Back
欢迎回来

55. The New Approach to Quality Assurance
Introducing the the new approach to quality assurance criteria
feedback

56. Criteria 4.2 4.2 Signed pre-delivery checklist.
The centre must provide documented evidence to ensure that assessments are valid, reliable, equitable and fair.
Signed pre-delivery checklist.
Standardisation meeting minutes, internal audit, review records

57. Criteria 2.3
2.3
Records must be maintained to provide evidence that the centre has sufficient competent staff who have the necessary qualifications, occupational experience and understanding to support the assessment and internal verification of qualifications being offered in the centre.
CVs/staff info sheets, CPD records, copies of relevant certificates, list of current assessors and internal verifiers, changes to the deployment of assessors/internal verifiers

58. Criteria 4.8
4.8
Outcomes of External quality assurance must be disseminated to appropriate staff and any action points must be monitored against agreed timescales.
Signed distribution list, corrective action log/report, minutes of meetings

59. Criteria 3.2
3.2
Candidates' development needs and prior achievements (where appropriate) must be matched against the requirements of the award. These must then be related to agreed personal action/training plans.
APL evidence, skills profile, training needs analysis, assessment plans, reviews

60. Criteria 4.6
4.6
The centre must comply with requests for access to premises, records, information, candidates and staff for the purpose of external quality assurance.
Documented procedure for handling QA visits, roles and responsibilities, site selection checklists, permission for SQA QA representatives to obtain access

61. Criteria 2.7 2.7 Current Health & Safety certificates
There must be evidence of initial and on-going reviews of accommodation, equipment and reference, learning and assessment materials.
Current Health & Safety certificates
itinerary checklists, procurement records, library contents, system for supporting e-assessment
Records of review

62. Criteria 4.3
4.3
Evidence of candidates' work must be accurately and consistently judged by assessors against SQA's requirements.
Assessor reports, records of achievement, standardisation meeting minutes, IV reports, Qualification Verification reports

63. Criteria 4.1
4.1
The centre's documented assessment and verification procedures must be implemented and the subsequent assessment/verification practices reviewed, the results of which must be recorded and actioned.
Minutes of assessor/internal verifier meetings, internal audit, review records

64. Criteria 4.7 4.7 Candidate evidence, documented retention policy
Candidate evidence must be retained in line with SQA requirements.
Candidate evidence, documented retention policy

65. Criteria 4.5
4.5
The centre must take steps to ensure that assessment evidence is the candidate's own work.
Candidate disclaimer, Candidate and staff induction materials: assessor/IV roles and responsibilities

66. Criteria 5.6
5.6
Comments/queries about the qualification specification, assessment guidance, qualification verification or related SQA matters must be resolved and recorded.
Correspondence file, s, standardisation meeting minutes, notes of action points and resolution

67. Preparing for Qualification Verification
The Qualification Verifier role activities
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

68. Making verification decisions
Deciding on the sufficiency of evidence can be likened to making an assessment decision
Making an assessment decision takes account of the overall strength of the evidence provided
One source of evidence will be sufficient at times. At other times, a number of sources will have to be considered, dependent on each centre’s approach to meeting quality assurance requirements and to some extent the structure of each criterion

69. One evidence source could be sufficient in some cases for this criterion:
Candidate’s development needs and prior achievements (where appropriate) must be matched against the requirements of the award. These must then be related to an agreed personal action/training plans.
The following are examples only, and do not constitute a
complete list of all possible types of evidence to support the
above criterion.
APL/RPL/APA evidence
Initial/diagnostic/formative assessment, skills profiles, training needs
analyses
Personal development/action plans, competence reviews, learner
agreement/contracts, assessment plans

70. More than one evidence source is likely to be considered for this criterion:
The documented assessment and verification procedures must be implemented and the subsequent assessment/verification practices reviewed, the results of which must be recorded and actioned. 
The following are examples only, and do not constitute a complete list
of all possible types of evidence to support the above criterion.
Records of assessment
Records of internal verification/audits, monitoring of assessor/verifier
practice
Reviews of assessment and verification practices

71. The main guiding principles for making verification decisions
Decisions should be based:
on the sufficiency of evidence relating to each criterion
only on the criteria provided
The evidence examples given in the guidance
document are just that. They are not a shopping
list of evidence sources that all centres must have.

72. Awareness of things that could influence on our decisions
Attitude of centre staff
Own
standards
Own
experience
Own
mindset (closed)
Our confide-ence
Previous centre perform-ance

73. Insufficient/no evidence = action points
Sufficient evidence - this means that the centre has provided evidence that fully meets the criterion (no action points required)
No evidence – this means that the centre has not provided any evidence in support of the criterion (action points required)
Insufficient evidence – this means the centre can provide some evidence in support of the criterion (action points required)

74. Agreeing action points
Is there anything wrong with this action point?
“You should submit evidence of having
disseminated the results of Qualification
verification visits to relevant staff”

75. Action points should be SMART
Specific: Centres should be clear in terms of the action they need to take to close-off the action point.
Measurable: How will SQA or the centre know that the action point has been met, what will be the measure of success?
Achievable: The centre must have sufficient time in which to achieve the agreed action. Please factor in time it takes for SQA to receive, process and edit the report and send it to the centre.
Relevant: The action must directly relate to the criterion
Time bound: Agree a specific and realistic.
date for the action to be completed by.

76. Questions

77. New Approach to Quality Assurance
Welcome Back
欢迎回来

78. The stages of Qualification Verification
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

79. The stages of communicating verification decisions
Plan to deliver balanced feedback
Explain your verification decisions

80. Plan to deliver balanced feedback
Plan to include:
Good practice
Recommendations
Action points (where appropriate)
Consider:
Location
Order of feedback and emphasis
Words and tone

81. Explaining your verification decisions
Where action points are given, ensure you:
have explored all possible evidence sources
gather all information to support your decisions
can justify your decisions - clearly relating them
to criteria requirements
Give the centre an opportunity to ask questions and clarify points

82. Communicating Decisions
Role play exercise:
David as a Qualification Verifier feeding back at the end of a Qualification Verification visit to Matthew who is the SQA Co-ordinator.
Please note your observations of the following on your observation sheet:
The content of the Feedback
Attitudes and behaviours of the Qualification Verifier and
the SQA Co-ordinator
Any possible improvements

83. The stages of Qualification Verification
Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report

84. The stages of completing the Qualification Verification Report
Ensure report content matches your feedback
Justify your verification decisions
Write clear action points
Check report prior to submitting to SQA

85. The Qualification Verification Visit Report

86. The Criteria Pages

87. The Required Action Log

88. The Summary Worksheet

89. Recommendations and Good Practice Worksheet

90. Writing Effective Verifier Reports
SMART action points
Report writing guidance available
ccc/Writing for SQA part A.pdf
Write report as soon after visit as possible
Checking content thoroughly

91. Assignment: Report Writing
Assignment will be ed to you by Thursday 20th September.
You will be asked to:
Review a Case Study
Complete a Qualification Verification Visit Report
Return to by Friday 12th October

92. Assignment: Report Writing
SQA will provide you with:
Guidance document on how to complete a qualification verification visit report
Feedback on each report to provide support
byFriday 26th October 2012

93. Summary To introduce you to SQA’s New Approach to Quality Assurance
To help you understand the key changes to Quality Assurance process
To help you become familiar with:
The new Quality Assurance Criteria
The new Confidence Statements
How visits are planned, conducted and reported

94. Questions