Evaluation Units Open Days Some experience of Implementing Agency in evaluating Pre-Accession Entrepreneurship Development Programmes Polish Agency for Enterprise Development Ex-post evaluation of the Phare 2000 ESC Regional Component HRD – case study and future activities in that area Jacek Pokorski Evaluation Section PAED Rome, Italy, July

Polish Agency for Enterprise Development in brief  PAED was established in the year  It is government agency responsible for implementing sectoral policy and cooperating with regional partners in managing implementation of regional programmes towards SMEs and HRD.  Scope of PAED interest:  Small and medium-size enterprises development  Export Promotion  Regional Development  Promotion of modern technologies  Job creation, human resources development and counteracting unemployment  Its activities are financed from the State Budget and European Union resources:  pre-accession programme Phare ESC  Structural Funds: - SOP Enhancement of Competitiveness of the Enterprises (European Regional Development Fund) - SOP Human Resources Development (European Social Fund)

Evaluation in PAED  PAED as the Implementing Agency for Phare ESC programmes was obliged to evaluate programmes after the end of its implementation process.  In 2004 an independent unit for evaluation matters was establish in the PAED – Evaluation Section  Activities of Evaluation Section:  designing, commissioning in tender procedure on-going and ex-post evaluation of programmes implemented by the PAED  disseminating the evaluation results  taking part in evaluations managed by the National Evaluation Unit  evaluation capacity building in the polish public administration area (organizing the First Evaluation Conference in Poland entitled: “Evaluation of the socio-economic programmes financed by structural funds” cooperation with universities and public institutions, organizing evaluation training courses for public administration)

Phare ESC  PHARE (Poland, Hungary - Assistance for Restructuring of their Economies) – the greatest assistance EU programme for the Central and Eastern Europe countries  From 1990 to 2000, under the framework of Phare programmes, Poland recived ca. 2,5 billion EUR  New Orientation for Phare (1997) – the Phare programmes had become an instrument for european integration. The EC extended an accountability of candidating countries in a scope of programming, implementing, monitoring and evaluation of projects co-financed from the Phare Fund.  Phare ESC (Economic and Social Cohesion) – regional and sectoral assistance to HRD, SMEs and Infrastructure Development

Phare 2000 ESC Regional Component HRD  Total budget: 30,7 milion EUR (75% Phare / 25% National Co- financing)  Time and location: November 2002 – February 2004 / 5 Polish voivodships (4 eastern regions with low economy development indicators and Silesia voivodship suffering from negative effects of industry restructuring process)  Types of projects:  Training and business counselling for unemployed and threatened by unemployment  Training for employees of SMEs  Assistance to development of local employment pacts  Specialised training for future employees of Tourist Information Network and tourist sector  Promotion of entrepreneurship in secondary schools  Objectives of Component HRD Increasing of human potential in regions including: increase of the employment possibility, development of entrepreneurship, increase of companies and thair employees adaptation capabilities, equalising of chances of men and women in the labour market

Ex-post evaluation of the Phare 2000 ESC Regional Component HRD  Objectives of evaluation  obtaining of increased knowledge on the effects of the programme PHARE 2000 Social and Economic Cohesion – Component Human Resources Development  better understanding of mechanisms contributing to obtain those effects with the aim of improvement preparation and implementation of the programs which will be financed from the means of the Phare and the ESF and will be implemented by the PAED  Evaluation criteria: effectiveness, efficiency, relevance, sustainability  Three perspectives of evaluation:  total assessment of the program  assessment of the HRD Component in terms of regions  assessment of particular types of pojects  Judgement technique: independent specialists assessment

Methodology of evaluation FIRST STAGE Initial stage - the analysis of the documentation connected with the program (desk research) SECOND STAGE Internal evaluation – 20 individual deepened interviews (IDI) with key people managing the projects (PAED, VLOs, conractors) THIRD STAGE Qualitative evaluation – 17 focused group interviews (FGI) with 96 beneficiaries of the projects (one FGI for every projects in each regions) FOURTH STAGE Quantitative evaluation – 1365 computer assisted telephone interviews (CATI) with beneficiaries of the projects (about CATIs for every projects in each voivodship)

A few main outcomes of evaluation

Employability of beneficiaries Training and business counselling for unemployed and threatened by unemployment Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

The average cost of creating one workplace (EUR) Training and business counselling for unemployed and threatened by unemployment Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Assistance to development of local employment pacts Training effects:  ASSUMPTION: signing 18 local pacts  ACHIEVEMENT: 54 local pacts signed Assumed and achieved number of pacts established Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Assistance to development of local employment pacts Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Problems and limitations of the Phare 2000 ESC Regional Component HRD evaluation  insufficient experts input in assessment process  difficulties of the evaluation team to work out useful and realistic recommendations (they had some problems with understanding the specific of public policy managing process)  limited possibilities of making comparisions:  the first program in Poland in such framework  the first evaluation ex-post in Poland of such scope  very extensive report:  evaluators had difficulties to assess and choose the priority issues (too much focusing on details)  the lack of possibility to construct comparision groups and estimate netto effects of the programme with utilizing data from public statistics systems

Current and future activities in that area...

Evaluations of successive editions of the Phare ESC Regional Component HRD  PHARE 2001 HRD 1) beneficiaries research after 6 months (from the end of participating in the programme) - introduction to evaluation  desk research (contractors reports and other programme documents)  6200 CATIs carried out by VLOs under the PAED supervision  16 IDIs with regional projects coordinators 2) evaluation ex-post after 12 months  emphasising on the qualitative methods  1 FGI with VLOs employees, 5 IDIs with regional projects coordinators in PAED  150 CATIs with start-ups-beneficiaries  115 CAPIs with entrepreneurs-beneficiaries  100 PAPIs with school manager-beneficiaries  Pilotage attempt at estimating netto effects with utilisation the propensity score matching method. Data of „untreated grup” from PULS Information System (the data base of unemployed, registered in PLO) INNOVATION APPROCH IN POLAND!

 PHARE 2002 HRD 1) beneficiaries research after 6 months – introduction to evaluation  desk research  3000 CATIs with beneficiaries, carried out by contractor 2) Evaluation ex-post after 12 months  emphasising on the qualitative methods  1 FGI with VLOs employees, 5 IDIs with regional projects coordinators in the PAED  150 CATIs with start-ups-beneficiaries  100 PAPIs with school manager-beneficiaris  Estimating netto effects and benchmarking the effects of the Phare 2002 HRD projects with previous editons  Implementing propensity score methode enables to assess the impact of the programme on beneficiaries after 6 and 12 months (from the end of participating in training courses)  Treated and untreated grup are going to take part in the evaluation research Evaluations of successive editions of the Phare ESC Regional Component HRD

Thank you for your attention For more information on the evalution in Poland (especially in the PAED), please contact Jacek Pokorski Polish Agency for Enterprise Development Ul. Panska 81/83 PL Warsaw