Update on Sector Reviews Cassy Taylor Associate Director (Vocational Qualifications)

Aim of sector reviews To identify whether qualifications, and the qualifications system are meeting the needs of employers and learners in a particular employment sector. Including the range and nature of qualifications assessment content including currency comparability efficiency Welsh-medium roles of bodies in the system

Sequence of reviews Health and Social Care (including childcare) – published July 2016 Construction and the Built Environment – to be published Feb 2018 ICT – to be published summer 2018 Engineering and Advanced Construction – to commence Oct 2018 To be followed by: Financial services Customer services and retail Travel and tourism Hospitality and catering

Scale of publicly-fundable Construction-related qualifications in Wales – 2015 certifications

Methodology Involvement of CITB Stakeholder advisory panel Interviews and discussion groups – 117+ employers, 70+ others Learner engagement – focus groups 940 learners Technical review – 23 qualifications, 14 reviewers International comparison work – Canada, New Zealand, Australia, Germany Online survey

Model for assessing effectiveness of qualifications Valid Reliable Manageable Engaging

CBE Findings 1 – The qualifications system System is non-strategic, complex, confusing and proliferated Progression routes neither clear nor sufficient – dead ends, repetition, gaps Specialisation too early - core, inter-personal and multi-skills are needed Apprenticeships too short Challenges in recruiting able learners Lack of information for learners, employers and parents Shared apprenticeships add value for learners and employers Learners on apprenticeships do benefit from learning programmes Learners feel well supported by their tutors

CBE Findings 2 – The qualifications Inflexible and not responsive to the needs of employers and the wider economy Content is sometimes outdated, irrelevant and/or insufficient Poor coverage of skills for traditional buildings Poor coverage of skills for new building techniques The balance of theory, practice and real work is not right Assessment low on validity, reliability, manageability and engagement Problems with ‘Levels’ Internal and External quality assurance of assessment inconsistent and/or poor Gaps in the technical knowledge and skills of the assessment workforce Limited assessment through the medium of Welsh.

CBE - Problems with assessment burden of assessment language, terminology and format of knowledge assessments validity of simulated assessment activities organisation and sufficiency of work-based assessment availability and validity of naturally occurring evidence inconsistency of assessment and quality assurance poor engagement with employers in the assessment process lack of differentiation in pass/fail qualifications availability and expertise of assessors availability of Welsh-medium assessment.

CBE - Problems with levels Significant repetition across levels 1, 2 and 3 Difference in performance between levels unclear/insignificant  Level of complexity across different trades not equivalent across the ‘same’ level Real progression within a skills area may be represented by broadening or deepening the learner’s skills and knowledge within a level rather than progression to the next level Equating level 2 vocational qualifications with GCSEs is unhelpful Some assumptions about autonomy = supervision  Mismatch between skills for work and cognitive skills for Higher Education.

Problems with Levels “We are actually assessing vocational students who have chosen a vocational route academically. We are possibly stopping those who could be the best trades people from progression simply because we are judging them on academic values.” Head of Construction in a Further Education College

CBE Proposals for Action – Phase 1 Ask awarding bodies to consider the findings about assessment Ask sector bodies, with awarding bodies, to review the standards for content Share findings on apprenticeship programmes with Welsh Government Share findings with Careers Wales Take findings into account in addressing reviews of Welsh Bacc and Essential Skills Consider wider issues about assessment including: levels in vocational qualifications the use of technology the manageability of competence-based assessment

CBE - Proposals for action – phase 2 14-16 Level 1 / 2 16-19 progression to HE Full-time FE Preparing for Apprenticeships Apprenticeships Develop approval criteria and invite submissions from awarding bodies Is it viable for awarding bodies to amend without a guarantee of being the sole supplier in Wales? Commission a broad-based Foundation qualification with trade-specific pathways. Commission a Progression qualification for Construction and for Building Services for those who don’t progress immediately to apprenticeships. Commission a generic apprenticeship qualification for CBE with: Pre-registration formative assessment by employer and employer sign-off (not part of qualification) Work-based project Professional discussion Online multi-choice knowledge test

Interviews and focus groups Online survey Technical Review Progress of the ICT Sector Review Interviews and focus groups 749 learner 63 employers 67 schools 9 colleges 7 universities Online survey 536 learners 79 learning providers 16 employers 14 others Technical Review Complete Additional work on new qualifications International Comparison Comparison of ICT in 11 nations

ICT - Feedback from employers Do not know what the various different qualifications cover Do not value the IT qualifications taken at 14-19 – they see these more as ‘user’ qualifications Learners lack conceptual understanding Skills/aptitude/experience more highly valued than qualifications Qualifications don’t give enough practical experience – most completely re-train for specific needs of business Technically capable candidates often don’t demonstrate necessary softer skills, such as customer service Learners need to develop problem solving and troubleshooting skills, while understanding processes. Some employers value vendor-specific qualifications and industry recognised certificates such as Comptia +

ICT - Feedback from learners in Further Education Those taking qualifications for progression to HE are not very engaged with the subject – see them as an ‘easy’ way to get grades for University Very low satisfaction with currency of content, across ICT qualifications Very low satisfaction with the hardware and software available Courses ‘not relevant to the working world’ Far too much ‘writing-up’ for assessments rather than doing practical tasks No opportunities to be creative Low confidence in tutors’ knowledge and ability Frustrations with additional mandatory qualifications

ICT - Feedback from teacher events Pre-occupied with having a GCSE and not a vocational qualification Questions about the relationship to the curriculum review High level of resistance to change Some acceptance that modernisation needed and a wide range of suggestions for new content Agreement that print-screen approaches to assessment were not engaging Highly variable learner starting points Variable teacher currency and flexibility Support for more assessment of product rather than just of process Some support for a more project-based approach, but stressed importance of external exams Benefits of online assessment recognised, but some severe resource constraints; and Would welcome more engagement with businesses, for example problem-setting, webinars, videos.

Everyday users/basic operators Types of roles Everyday users/basic operators Operators / informed users – using corporate systems Technological Solutions Interface between users/ operators and maintainers/ creators Maintainers Networks, equipment Creators Makers Current 14-19 provision Digital/ IT user skills THE GAP? Digital solutions Computer Science Approach of the Review Review Consider Developing – or do the recent new qualifications address this? Light touch review

Summary of underpinning challenges Variable learner starting points Variable (and lacking) teacher skills Variable resources Wide range of potential content Pace of change

Early thoughts on way forward Recently introduced qualifications may address some of ‘gap’ issues at 14-16 Project-based approaches to assessment most flexible to address other variabilities Flexible approaches to content Identify how to assess digital competence elsewhere Need creative solutions to address teacher workforce and resources. To be continued!