Reflective Evaluation About existing evaluation and self- evaluation instruments and reasons for inventing something new for CVT Irina Michel, Gerald Heidegger, Wiebke Petersen

Reflective Evaluation Transcultural aspects  Copenhagen Process:  No „unification“ of systems in VET and CVT but subsidiarity principle (contrast to Bologna process for higher education)  Reason: VET and CVT systems too strongly embedded in cultural and economic conditions  Therefore reduction of the aims to the following four fields:

Reflective Evaluation  With respect to our project:  European Dimension  ---- transcultural mutual learning  Accreditation of non-formal and informal learning  ---- accreditation of CVT which includes non-formal learning  Quality (of processes and outcomes)  ---- our new tool aims at self reliant quality assurance  Transparency  ---- the criteria for the self-evaluation should be transculturally transparent

Reflective Evaluation  Transcultural comparison: Four main types of -Welfare system -VET and CVT systems -Re-Integration schemes See overhead transparency

Aspects of the history of quality management and its tranfer to the educational sector - The idea of quality management stems from production industry (ISO 9000) -During the last decade the idea of quality management has been broadly transfered to non- profit organisations (EFQM – European Foundation for Quality Management) -The first non-profit sector that has taken over quality management was the care sector – soon stressing self-evaluation (because a lack of „hard“ outcomes) -Today a lot of public schools go through evaluation processes. -For disadvantaged young people: Our method QSED (Quality through Self-Evaluation and Development)

Figure 4.1: Contexts of evaluation Contexts of evaluation common understanding of the phenomenon that should be evaluated historical and cultural embeddedness of the phenomenon of evaluation national annd international evaluation discussions Output indicators or personal development From ISO 9000 via EFQM to QSED different evaluation interests, fears and, targets

Carried out by OECD, CEDEFOP Evaluation trends Carried out by IDEA, IOCE Indicator centered evaluation innovative evaluation approaches targeting empowerment or participation strict, highly structured procedures Measuring soft outcomes

Purpose and meaning of evaluation Evaluatio n practices and methods Context of evaluatio n Expertise of evaluatio n Target/ Object of evaluation Theoretical interests in evaluation and meta- evaluation relevant aspects of evaluation

External evaluation  Advantages  Seemingly „Objective „, Reliable Outcomes  Easily transferable results  No self-deception  No cheating(???)  Should be combined with (internal) self- evaluation  In order to….  Disadvantages  To measure „soft“ outcomes is very difficult/impossible  Processes are aiming at the targets of the external evaluation  In this way processes are severely narrowed down  Practitioners are objects of external powers  They become disempowered  They become passive  avoid these

staff orientation (3) staff contentedness (7) processes (5) societal responsibility and image (8) resources (4) customer contentedness /satisfaction (6) company results (9) criteria of EFQM politic and strategy (2) leadership (1) Figure 6.1: Criteria of EFQM Existing European evaluation approaches: example EFQM

European standard: the EFQM Example for the criteria  Processes  This criterion is defined as follows: How does the institution recognise, identify, carry through tests and improve its processes?  It is divided into five subsections which deal with the following questions and for which several starting points are given:  5a ”How are processes identified which are significant for the company results?”  Starting points could be, how the institution...  …defines core and support processes (1)  …identifies core and support processes(2)  …values repercussion on the business results(3)

European standard: the EFQM  Using the EFQM as an evaluation tool  In the first step it takes place through self-evaluation  The second step includes the coming in of external evaluators  Underlying assumptions and targets of evaluation according to EFQM  “Is the client content with the product which is given?”.- Is it the “Customer”, that is the Labour office???  Advantages of the EFQM in comparison with other evaluation tools:  it tries to take care also of soft outcomes

Adaptation of the EFQM model to a re-integration institution for disadvantaged youngsters Benchmark ing continuous improveme nt customer orientation Self directing JAW-stamp of quality Figure 6.2: JAW-stamp of quality client orientation

A Self-Evaluation instrument for schools: A different idea of quality  Basic ideas of Q2E : Quality though Evaluation and Development („Entwicklung“) with a focus on self- evaluation client orientation awareness of the own quality promise optimizing through error recognition aspects of the new quality understanding Figure 6.3: Aspects of the new quality understanding

Example of a criterion in Q2E: “Shaping of teaching and learning processes (methodical-didactical arrangements )” - to be ticked from 1(weak) to 4 (strong) - 1)The teacher attaches importance to the goal that aims and intentions of the lessons are understood by the pupils. The learners see the importance of learning aims and contents. 2)The teacher arrives at explaining complex learning processes and difficult facts in relation to experiences and knowledge of learners. 3)The teacher arrives at initiating the pupils´ interest for the contents and engaging them for participation.

Example of a criterion in Q2E: “Shaping of teaching and learning processes(methodical-didactical arrangements )” - to be ticked from 1(weak) to 4 (strong) - ) 4)Space of active participation and self-directed learning of the pupils in the lessons is offered 5)The teacher arrives at making the pupils aware of their responsibility for their learning and he supports this through adequate measures. 6)The teaching is shaped in a way that pupils could create a strong relation between theory and their own experiences.

… basic instrument for focus evaluation …as basis for the development of a quality handbook … as basis for a quality comparison between schools...basic instrument for the development of a school specific evaluation instrument … help for the development of a school specific profile of strengths and weaknesses … basic instrument for the development of a quality model Q2E can be used as... Figure 6.4: Options for using Q2E

Our new IT-Instrument: QSED  Centred on self-evaluation of practitioners  To be adapted to the specific circumstances  Culturally adaptable  QSED („Quality through Self-Evaluation and Development“) for support schemes for disadvantaged young people should now be adapted to CVT

Reflective Evaluation Target group of the instrument: Trainers and practitioners for  People at risk in the labour market:  - Unemployed, often long term  - People under threat of being fired  - Women returners  - But also: People in conventional CVT courses

Some features of the QSED („old tool“)  Six dimensions  Three levels  Highly(!!!) interactive IT-Tool  Adaptation through the users themselves  The users can completely change the instrument apart from the basic structure (dimensions and levels)  Next: the dimensions

QSE D Self-Evaluation R and R eflection C ollaborative networks of actors Recognition of A skills/ A ssessment F unding/ Administrative structures I nclusiveness S ituated Learning to the end

Transformation of QSED for the new target group  Some dimensions: the same, although with new aspects and new criteria  Some dimensions: new, for the specific needs of the target group (both participants and practitioners)  Next page: tentative outline of the dimensions for the new tool:  „Developing quality in CVT“

“DQCVT” New! Self-Evaluation R and R eflection C ollaborative networks of actors Validation and accreditation of competencies New! F unding/ Administrative structures Life situation/ Experiences New! Didactical/ methodological Concepts New! to the end

Transformation of QSED for the new target group  Starting from QSED:  Three levels („Multi-level approach“): Explanation: next page  Macro: „Structural“  Meso: „Institutional“  Micro: „Individual“  Also some other explanations

Structural-level: o n this level you find statements which point out power relations in society. Institutional level: On this level you find questions which deal with the activities and the processes within Reintegration programmes. Individual level: On this level you find questions which deal with the actions and experiences of practitioners and learners. Explanation: Here you find further information regarding the statements and questions. Suggestions: Here you can write in suggestions for influencing the societal, economic and political conditions. Answer and Change: Here you can answer or change questions in order to design your own tool. Using this button you get back to the cluster which shows the dimensions. Using this button you get back to the content of the dimension. Back Back to content Back to Index

Developing Quality in CVT  Now let´s try out and look at the „old“ QSED in order to get some ideas about the  New  DQCVT