2. Monitoring SQA Qualifications over time Dr Rob van Krieken

3. Another bridge in Arnhem

5. Content  Why is SQA monitoring over time?  What are the standards SQA is monitoring?  SQA’s monitoring programme  Implementation  Reports and actions

6. Why is SQA monitoring over time? to avoid  ‘creep’ within a stable system  shock when the system changes

7. Example of “creep” Year 1Year 2 Cut-off% PassCut-off% Pass 4986%4987% 5080 %5081 % 5172%5173%

8. Example of “creep” Year 2Year 3 Cut-off% PassCut-off% Pass 4987%4988% 5081 %5082 % 5173%5174%

9. Example of “creep” Year 3Year 4 Cut-off% PassCut-off% Pass 4988%4989% 5082 %5083% 5174%5175%

10. Example of “creep” Year 1Year 4 Cut-off% PassCut-off% Pass 4986%4989% 5080 %5083% 5172%5175%

11. System changes  Vocational qualifications are reviewed systematically at the end of their duration  O-grade to Standard Grade (1984 – 1990s)  SCE Higher – Revised Higher (1992 - )  (Revised) Higher - Higher (Still) (2000)  CSYS – Advanced Higher (2001)  New National Qualifications (2014-2016)

12. What are the standards SQA is monitoring?  ‘Minimum level of demand’ –Laid down in specifications (“standards”) –Exemplified in assessments/marking/boundary  Within the educational context of –SCQF levels, similar qualifications –Quality and quantity of teaching and learning –Cohort

13. Monitoring programmes  Different programmes for National Courses and Vocational Qualifications (HN and SVQ)  Sampling by subject area and level  Cycle depends on qualification / level

14. Sample Higher National Diplomas 201420152016201720182019  Beauty Therapy  Administration and Information Technology  Early Education and Childcare: Childhood Practice  Social Care  Social Service  Hospitality Management  Computing: Software Development  Travel and Tourism  Nautical Science  Early Education and Childcare  Beauty Therapy  Administration and Information Technology  Social Care  Hospitality Management  Computing: Software Development  Social Service  Travel and Tourism  Nautical Science C&A Collect and archive C&C Collect and Compare

15. Sample Higher National Qualifications 20132014201520162017201820192020 English C&A C&C Mathematics C&A C&C Art and DesignC&AC&C Business ManagementC&C Chemistry C&A C&C ComputingC&C Engineering Science C&C FrenchC&AC&C GeographyC&AC&C RMPSC&A C&C

16. Implementation: Panels, materials, and tasks  Panel: –3 subject experts  Materials: –Specification –Assessments, Marking Instructions –Evidence –Judgements  Task: –Questionnaire

17. Questionnaire 1Educational Context: Are there any general changes that may have influenced the general ability of candidates in one of the years, in particular the way they meet the demands made by the syllabus and exam?

18. Questionnaire 2Unit Specification or Standards and Guidance 2.1Are there differences in the Unit Specifications and Guidance that affect the level of demand? 2.2Were the Unit Specification Standards in 2011 on the whole more / no more / less demanding than before?

19. Questionnaire 3Assessment 3.1Are there differences in Assessment practice that affect the level of demand? Please tick the relevant boxes and add if necessary. 3.2Was the Assessment in 2011 on the whole more / no more / less demanding than before?

20. Questionnaire 4Quality and Judgement of Evidence 4.1 Are there differences in the quality of the evidence and in the way this has been judged? Please tick the relevant boxes and add if necessary.

21. Questionnaire 5Overall judgement A reasoned overall judgement of the level of demand of the qualification as a whole, taking into account standards, assessments, and evidence.

22. Reports and actions  Overall report  Summary of conclusions  Actions SQA undertakes

23. Issues and answers 1  Too much change  Overall judgement  Quantity and type of evidence  Panel membership  No looking Backward