Guide on VET providers’ self-assessment/self- evaluation by using the EQAVET indicators Working Group on Indicators Phase II - Meeting 2 Brussels, 21st September 2011 Prepared by Maria Emília Galvão

Charting a course in VET Quality Assurance: from internal to future QA 1st wave (1970s): Internal Quality Assurance [...] mainly refers to the achievement of planned education goals particularly in terms of students’ education outcomes. 2nd wave (1980s): Interface Quality Assurance […] education quality and its assurance is put into […] satisfying the needs of multiple stakeholders and ensuring the accountability of education to the public are crucial criteria for determining education quality. 3rd wave (1990s-) Future Quality Assurance […] will emphasize the development of students’ contextualized multiple intelligences […] and the processes of triplization […] in education. Source: CHENG, Yin Cheong, Paradigm Shifts in Quality Improvement in Education: Three Waves for the Future, (2001) Plenary Speech Presented at the International Forum on Quality Education for the Twenty-first Century, Co-organized by UNESCO-PROAP, China

AssessmentEvaluation Timing Formative, i.e. ongoing to improve Summative, i.e. final to gauge quality Focus of measurement Process- OrientedProduct- Oriented Relationship between administrator and recipient Reflective (internally defined goals/criteria) Prescriptive (externally imposed standards) Findings and uses Diagnostic (areas for improvement) Judgmental ( by comparison to similar objects) Modifiability of criteria/measures FlexibleFixed Standards of measurement Absolute, i.e. individual Comparative Relation between objects of assessment/evaluation CooperativeCompetitive Assessment & Evaluation Source: Apple, D.K. & Krumsieg. K. (2008) Process education teaching institute handbook. Pacific Crest.

Assessement & Evaluation ( cont.) Source: Apple, D.K. & Krumsieg. K. (2008) Process education teaching institute handbook. Pacific Crest. Both processes: Require setting and using criteria Use measures to identify the level of performance Are driven by evidence Need accurate and holistic data for an accurate summary Provide motivation Require reporting to the performer

Assessment & Evaluation in a nutshell… Source: Apple, D.K. & Krumsieg. K. (2008) Process education teaching institute handbook. Pacific Crest. Assessment is the term used to look at how the level of quality of a performance or outcome could be improved in the future; it includes strengths that should be sustained as well as high-priority areas for improvement. The assessment process is not concerned with the level of quality; only with how to improve the level of quality. Evaluation is the term used to describe the determination of the level of quality. The evaluation process focuses only on the actual level of quality with no interest in why that level was attained.

Perspectives on VET Evaluation ( Adapted from MacBeath, 2002 ) Accountability perspective Knowledge perspective Developmental perspective Purpose To provide data on performance, effectiveness and value for money To generate new insights about quality of VET To strengthen VET provider’s capacity for self -improvementt Audience Relevant stakeholders, e.g. national authorities, Management, teachers/ trainers VET provider, teachers/ trainers, learners, employers Provider VET provider management Teachers /trainers, learners, management Management, teachers/trainers, learners, employers Internal/ External relationship Summative; External evaluation supported by data from self - evaluation Diagnostic, primarily through self - evaluation Formative, primarily self -evaluation with external critical friend’s support.

Overview of a self –assessment /evaluation process Steps in the processDecision making 1. Specify, select VET goals and evaluation objectives What is the general focus of the evaluation? 2. Establish standards/ criteriaWhat benchmarks/ measures will be used ? 3. Plan evaluation designWhat key questions need answering? What is feasible in terms of budget, available resources, time & expertise? 4. Select /develop data gathering method 5.Collect relevant data What data will be gathered? From whom, by whom and how will data be gathered?

Overview of a self –assessment/ evaluation process (cont.) Steps in the processDecision making 6. Process and analyse relevant data 7. Compare data with evaluation criteria ( see Step 2) How will data be analysed and interpreted? Who will analyse and interpret data? 8. Report and feedback resultsHow, when and to whom will results be communicated? 9.Review of objectives and /or procedures for change How will evidence be used to improve future performance? 10.Throughout steps 1-9How will the self-evaluation be managed ( i.e. allocation of tasks, staff to be involved, etc)?

Self- assessment/evaluation: main elements for an effective process VET provider’s self-evaluation is mainly effective when it  identifies and responds well to the expectations and needs of learners, employers and other relevant stakeholders;  sets demanding objectives as part of its review plans;  benchmarks its performance in order to act on outcomes, either for improvement or sustainability reasons.

Definition of Quality Objectives ( Planning) Measuring & Reporting Quality (Monitoring) Quality Improvement (Review) Components of the European Quality Assurance in VET QA: practice of actively using information to improve VET provision and/ or VET (sub) system

Definition of Quality objectives to improve VET (sub) systems & provision. Policies and procedures are developed to define how the organization should operate in order to reach these objectives. Quality measurement and reporting: use of data to measure progress in meeting objectives; processing of data into a report that is easy to understand and use; distribution and presentation of these reports to people who can best make use of them. Quality improvement: use of performance reports to make changes needed in order to meet quality objectives. QI may a) involve changes in VET policies, procedures, programmes or staff training b) be an incremental or a breakthrough improvement. Components of Quality Assurance (QA) in VET ( cont.)

Developing a QA culture based on self – assessment/evaluation by using the EQAVET indicators Why do the self- evaluation?  Evaluation topic(s)  Background and rationale  Evaluation objectives  Users of the evaluation What is the VET provider evaluating?  Key areas  Evaluation questions  EQAVET indicators How will the VET provider evaluate ?  Evaluation design  Data sources and procedures  Data analysis procedures How will the evaluation be managed ?  Evaluation activities and schedule  Evaluation team and level of responsibility  Resources ( logistical, administrative, financial).

Examples of scenarios for building QA systems based on EQAVET Scenario 1: no QA system for VET is in place A whole-system EQAVET implementation plan is needed: Base-line study, i.e. an analysis describing the situation prior to the intervention, against which progress can be assessed or comparisons made. Adopt EQAVET indicators and adapt them to local circumstances Identify resources and train/orient mid-level supervisors and front-line staff to the implementation plan; Develop a strategy for the presentation of findings, for review by management, for the ongoing updating of objectives and indicators and, if that may be the case, for linking of results to incentives.

Examples of scenarios for building QA systems based on EQAVET ( cont.) Scenario 2: a QA system has been in place (objectives, programmes and procedures have become outdated) To reestablish a QA approach and make it compatible with EQAVET, the following steps can be taken: Conduct a review of existing objectives, indicators, programmes and procedures to identify those that are out-of -date, irrelevant or causing problems or difficulties; Revise which of those need to be changed and rewrite them by adapting the EQAVET indicators to local circumstances; Review the data gathering process to be sure that data collectors are sufficiently trained, objective in their observations, and accountable for collecting data and conducting planning meetings and submitting reports on time.

Illustration of a table format for data gathering and analysis EQAVET Indicator Where will data be collected? How will data be collected? When and how often will data be collected? Who will collect data? What type of data analysis will be used? Who will report the Information? No.1 No. 2 No.3 No.4 No.5 ETC

Source:

Source: QUESTIONS? Question mark (Polygonia interrogationis)