1. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation Twinning versus Technical EU Pre-accession Assistance Evaluation Twinning versus Technical EU Pre-accession Assistance The consortium led by COWI/AS was awarded this assignment by DG Enlargement, Operational Audit and Evaluation Unit. Within the consortium ECORYS Nederland been responsible for implementation of the evaluation Presentation of Key Findings and Recommendations Twining National Contact Points' meeting Brussels, 16 June 2011 Pedro Andreo. ELARG E4, Operational Audit & Evaluation Presentation of Key Findings and Recommendations Twining National Contact Points' meeting Brussels, 16 June 2011 Pedro Andreo. ELARG E4, Operational Audit & Evaluation

2. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Outline 1.Approach 2.Key findings 3.Key recommendations

3. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation object and scope Comparison of delivery instruments: twinning and Technical Assistance* as provided under IPA 2007-2008, PHARE 2005- 2006, CARDS 2005-2006 and the Turkey pre-accession assistance instrument 2005-2006 *) TA concluded by means of restricted international tender procedures

4. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation objectives To assess the contribution of twinning and Technical Assistance instruments to the overall accession process in the Western Balkans and Turkey To provide lessons learned and recommendations for programming and implementation of future assistance, including through the development of cost-benefit assessment templates.

5. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Three main evaluation questions Is the instrument (TA or twinning) selection done appropriately? To what extent does the selection result in effective and efficient project implementation with a view on sustainability? How can the selection process be improved (recommendations)?

6. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation approach Focus - Three levels Broad analysis - focus: Entire twinning and TA effort in the region and reconstruction of the decision process for selection of instrument Sector analysis - focus: Selected sectors (Agriculture & Fisheries, Energy, Internal Market and Economic Criteria, Justice and Home Affairs and Finance) Project analysis: Pairs of comparable projects, examples of projects, beneficiaries and providers having worked with both instruments.

7. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation approach Tools 154 stakeholders were interviewed in Brussels (DG-ELARG), the EU Member States (NCPs, ex-RTAs), and in all candidates and potential candidates involved (all EUDs, all NIPAC offices, three CFCUs, altogether 35 beneficiary organisations) between March and September 2010. Three web-surveys resulted in 165 validly filled in response sheets from beneficiary organisations, twinning providers and TA providers

8. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ1: Appropriate selection of instruments? (1) In practice choice is based on three criteria: -Nature of assignment (acquis related or not) -Maturity of beneficiary institution -Capacity of beneficiary organisation No formal CBAs are performed. Perceived cost differences play only a limited role in the selection process

9. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ1: Appropriate selection of instruments? (2) Selection done by beneficiary, in a dialogue with EUDs, DG- ELARG. Overall beneficiaries are now well placed to make an informed selection between TA and twinning, although in Albania, Bosnia and Herzegovina, Kosovo (under UNSCR 1244/99) and the Former Yugoslav Republic of Macedonia several beneficiary organisations indicated that they still have difficulties in making a proper selection.

10. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ1: Appropriate selection of instruments? (3) TA as instrument still outnumbers twinning significantly, however in some sectors the number of twinning projects is close to half of the number of projects: Agriculture and Fisheries, Justice and Home Affairs and Finance The share of twinning projects increases over the years, and is larger in candidates than in potential candidates, both in number of projects (448 TA, 181 TW period 2005-08) and in terms of budget allocated (800 TA, 190 TW period 2005-08). Croatia has most experience with twinning.

11. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ1: Appropriate selection of instruments? (4) Perceived intangible benefits of twinning: The desire for establishing a relationship with a comparable organisation in the EU is an important driver for beneficiary organisations to select twinning The fact that twinning introduces a different working culture is less prominent as a reason for selecting twinning.

12. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ1: Appropriate selection of instruments? (5) Although twinning, when certain conditions are fulfilled, is definitely the preferred option, the following should be kept in mind: Acquis knowledge is not exclusively available within twinning providers Use of retired civil servants as RTA, however, may take away the intangible benefit of the lasting relationship Some mandated bodies may not have the proper profile.

13. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (1) No significant difference could be observed in the achievement of results in projects applying TA or twinning. Likely causes: Generally, the selection process has been effective In cases where for acquis-related assignments TA had been selected, the reasons for choosing TA were related to the limited capacity of the beneficiary. This appears to be in those cases an appropriate choice TA providers can also provide relevant acquis knowledge. The important effect of lasting relationship with an EU MS partner however is unique for twinning.

14. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (2) Strong points twinning most often mentioned: Appropriate knowledge Fostering relationship with EU MS organisation Contributes to change in organisational culture. Strong points of TA: Appropriate knowledge Flexible and controllable (steerable).

15. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (3) Weak points of TW: Long preparation stage Limited control options for beneficiary Administrative burden on beneficiary Weak points of TA: No mandatory results Price Sometimes too supply-driven.

16. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (5) Cost-comparison: On a unit-base (i.e. fees) twinning is on average cheaper than TA ( 46,619/Month for TW compared to 58,493/Month for TA. i.e. 23% higher for TA – exceptionally high cost cases excluded) the variance in price among TA projects is much larger. some 40% of TA projects are on a monthly base cheaper than twinning.

17. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (6) Main factors affecting the performance of both twinning and TA projects include: Realistic objectives Ownership by beneficiary; provider should not impose solutions Provider has understanding of background and environment beneficiary (this makes new EU Member States very appreciated twinning providers).

18. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key Recommendations (1) How can the selection process be improved? The commonly applied selection criterion, i.e. twinning is only suitable for acquis related assignments in acquis related beneficiary organisations that have sufficient capacity to absorb twinning. In all other assignments technical assistance may be more effective and efficient is practical and realistic. There is no reason to divert from this.

19. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key Recommendations (2) selecting either TA or twinning Specifying maturity of the beneficiary organisation (criteria): Is the organisation already legally established? Has role and mandate been laid down in legislation? Has the organisation made a decision on the manner in which it intends to realise the acquis? Is there political and/or public support for the above? Is management of the beneficiary organisation stable? Does the organisation avail of sufficient capacity (staff, space, IT etc) to absorb twinning? If a majority of these conditions are not met: opt for TA The intangible benefit of a lasting relation with a member state is key

20. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key Recommendations (3) for selecting a provider Insisting on more detailed twinning proposals would facilitate the selection of an appropriate twinning provider Demanding a personal presentation from TA providers, as part of the tender procedure, would facilitate the selection of an appropriate TA provider

21. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key Recommendations (4) for selecting a Tw provider It should be ascertained that: Mandated bodies considered as twinning providers are really part of the public administration; and that they are also able to secure access to relevant public bodies and relevant expertise in their own country during and after the project. The accreditation of mandated bodies as twinning providers is the prerogative of the respective EU member states. The beneficiaries may however use above conditions to select the most appropriate provider.

22. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Key Recommendations (5) for implementation of twinning Attention should be given to more flexibility in the execution of twinning contracts Preparation time of twinning projects should still be shortened

23. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit The Blue Print A blue print to support the selection of TW-TA was proposed as part of this evaluation

24. European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Thank you for your attention! Pedro Andreo. Head of Sector, Evaluation. ELARG E4, Operational Audit & Evaluation