Gilles Guillard, Markus Nussbaum Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum

The European Union Commission Economic & Social Committee Council Parliament Court of Justice Court of Auditors Commission Economic & Social Committee Committee of the Regions Ombudsman Data Protection Supervisor

The EU Institutions Posts in 2012 European Parliament 6655 Council of the European Union 3153 European Commission 25478 Court of Justice 1952 European Court of Auditors 887 European Economic and Social Committee 724 Committee of the Regions 531 European Ombudsman 66 European Data Protection Supervisor 43 European External Action Service 1670 Total 41159

Entry Streams Administrators (AD) e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers … Qualification: University degree (bachelor) Assistants (AST) e.g. secretaries, HR assistants, financial assistants, conference technicians … Qualification: Secondary education (at least) and relevant professional experience

linguistic skills test - Selection in Cycles AST Registration + Self-assessment CBT Assessment Recruitment Self-assessment Registration + CBT Assessment Centre Recruitment AD December March June September Registration + Self-assessment CBT / professional linguistic skills test - CV sift Recruitment Assessment Centre ADL

The AD Cycle 2012 Competition launched in March 2012 45000+ candidates for 296 posts 5 fields: Audit, EU Public Administration, Communication, Law, External Relations

Computer Based Tests Verbal reasoning 20 items from a database, 35 min, 23 languages Numerical reasoning 10 Items from a database, 20 min, 23 languages Abstract reasoning 10 Items in fixed test forms, 10 min Situational Judgement 20 Items in fixed test forms, 30 min, EN/DE/FR

Verbal Reasoning

Numerical Reasoning

Abstract Reasoning

Situational Judgement X X

Results EU Public Admin Audit Communication Law External Rel Sum AD5 validated application 12287 7892 2675 1389 3467 2023 4636 2130 6519 2338 45356 booked CBT 9557 6076 2182 1005 2525 1430 3615 1592 5016 1706 34704 participated CBT 8323 5336 1942 893 2132 1263 3113 1330 4412 1470 30214 reached CBT pass-marks 5502 3767 1427 603 1253 723 2076 830 2935 948 20064 succeeded CBT 190 102 108 34 77 51 144 43 64 21 834

Score Distribution - VR

Score Distribution - NR

Score Distribution - AR

Situational Judgement

Intercorrelations Verbal Reasoning Numerical Reasoning Abstract Reasoning SJT Analysis & Problem Solving SJT Planning & Organising SJT Quality & Results SJT Resilience SJT Working with others - .41 .29 .16 .23 .15 .17 .18 .32 .20 .14 .12 .40 .33 .31 .35

How EPSO monitors the behaviour of items Item Analysis How EPSO monitors the behaviour of items

Item Analysis Based on the dicotomous / polytomous Rasch model Conducted after every major competition Calibrate items and monitor their behaviour Making psychometrics as vivid as possible Georg Rasch (1901-1980)

Example: Cumulative Analysis VR+NR Test Responses Candidates Items VR 2 295 360 114 768 5 071 NR 1 147 680 3 490 Total 3 443 040 8 561 Oct 2012

Item Difficulty Adjustment Across Languages (IDAAL) How EPSO ensures fair opportunities for all candidates although testing in 23 different languages

Aligned Averages Algorithm (AAA) 1 2 3 4 -4 -3 -2 -1 5 1 2 3 4 -4 -3 -2 -1 5 EN A EN A -4 -3 -2 -1 +1 +2 +3 +4

Aligned Averages Algorithm (AAA) citem = common item in en and lang

Minimum Meansquare Method (MMM) 1 2 3 4 -4 -3 -2 -1 5 1 2 3 4 -4 -3 -2 -1 5 EN C A B EN A B C

Minimum Meansquare Method (MMM)

Person Parameter Contrast Measure (PPCM) 1 2 3 4 -4 -3 -2 -1 5 Cdts EN Cdts A

Person Parameter Contrast Measure (PPCM)

Item Difficulty Adjustment Across Languages (IDAAL) ... is a combination of these three approaches: Aligned Averages Algorithm (AAA) Minimum Meansquare Method (MMM) Person Parameter Contrast Measure (PPCM)

How to assemble test forms EPSO Rasch Item Combination Algorithm (ERICA)

Framework Candidates have to be treated equally: Same number of items Same difficulty of test form ➾ Adaptive testing is not possible EPSO has developed an algorithm involving two steps: Set up a "master test form" Create test forms corresponding to master test form

Target (quality criteria) Actual test forms correspond to master test form Limited overlap between test forms No over-exposed items Further improving gender balance

Defining the master test form Master test form difficulty: 5 easy difficult Master test form difficulty: 8 Master test form difficulty: 2

Weighting Items are approximately standard normal distributed Most items are around the average (0) available Weighting was implemented to better take account of item availabilities

Sample Test Form

Thank you very much for your attention! Any questions?