1. Impact evaluation of ALMPs in Macedonia: results and policy implications
Dr. Nikica Mojsoska Blazevski
University American College-Skopje

2. Why impact evaluation?
Measure effectiveness and efficiency of the ALMPs
Inform policymakers – evidence-based policymaking
Currently, main focus on immediate outcomes (% of employed workers at program end)
It fails to account for several possible effects:
long-term effects
substitution effect
Dead-weight loss, etc.

3. Why impact evaluation? (cont.)
Impact evaluation – actual impact of the program; implementation of “experiment” in social sciences
- Hypothetically, answer the question: what would have been the employment status of an unemployed person if s/he participates in a ALMP vs. if s/he did not participate
- For each treated individual, we look for the one individual among non-participants who is the closest neighbour/best match (observable and non-observable characteristics).

4. Types of programs Program Description Eligibility
Self-employment program
The program incorporates basic training in entrepreneurship, preparation of business plan, support for registration of a business and subsidy for starting a business. Unemployed receive support of 188,057 MKD of which 153,000 MKD for raw materials and business equipment, 13,500 MKD for covering social insurance contributions for 3 months and the remaining 16,897 MKD are used for the entrepreneurship and business plan writing training.
Young unemployed (up to 27 years of age), females unemployed at least 1 year, unemployed for more than 2 years
Internship 2010
Internship in companies, lasting 3 months. The cost per participant is 5,000 MKD (including personal income tax and insurance for professional health and safety).
Young persons, up to 27 years, with at least secondary education
Internship 2012
Same program, but obligation imposed to companies to employ at least 50% of the trainees, for a period od 12 months.
Training in deficient occupations
Competency-based training organized by a training provider in skills/occupations which are in shortage on the market. The curricula for the training is prepared by the VET Centre and approved by the Ministry of Education and Science. Lasts 3 months with additional one month of enterprise-based practical training. Participants receive 4,550 MKD per month as food and transport allowance (including personal income tax and insurance for professional health and safety).
Registered unemployed (with unemployment spell of min. 6 months) with similar occupations to those that are involved in the measure. Young unemployed persons (up to 27 years), registered for more than 3 months.

5. Types of programs (2) Program Description Eligibility
Training for known employer 2010
Competency-based training organized in a partner enterprise. Lasts up to 3 months. There is obligation on the enterprise to retain at least 70% of the trainees who complete the training, and not to reduce the total number of employees in a year. Participants receive 4,400 MKD per month, whereas the employer receives 3,000 MKD per trainee, as co-financing for the costs of the training.
No restrictions, all registered unemployed are eligible. The selection is done from those registered that have compatible education/occupation to those demanded by the employer.
Training for known employer 2012
Same as previous, but the obligation of the employer has been reduced to employing 50% of the program participants who complete the training.
Same
Wage subsidies
Wage subsidies for employment of 16,000 MKD per month covering wage, personal income tax and social contributions. Subsidies last 6 months with an employer obligation to keep workers on job for additional 12 months.
Older workers (55-64), young unemployed (up to 27 years of age), redundant workers,

6. Outcome indicators Current employment (ILO definition)
Non-employment-at-any-time
Employed at program’s end but currently unemployed
Inactivity
Hourly wage
Changes in the prospects of employment after program participation /cut-off point
Changes in financial status after program participation/cut-off point
Subjective labour market status

7. Self-employment: Findings
The grant for starting a business did not make a significant difference in the LM outcomes, only in the subjective well-being
The only positive effect is on inactivity: 30 p.p. lower chances to be inactive today
29 p.p. lower chances for participants to be employed at program end, but unemployed now
Effects on well-being:
- 29 p.p. higher subjective employment of participants
- Reported better chances for employment after program end

8. Self-employment: Findings (cont.)
If individuals are paired only based on observables, then the impact evaluation will show significant positive effect
Persons self-select into the program based on their unobservables (motivation, courage, entrepreneurial spirit, etc.)
In the current design, the program is likely to entail dead-weight loss

9. Self-employment: Suggestions
As a general recommendation, we propose changing the eligibility rules and selection procedure, rather than the penalties for the non-compliers;
The training program could contain a component related to business management and technical skills;
The quality of the training component should be monitored;
The size of the program (i.e. number of program participants) could be reduced whereas the amount of the grant increased;
The training could be designed so that it does not necessarily lead to a grant but may help participants to gain access to commercial credits;

10. Self-employment: Suggestions (cont.)
6. The amount of the grant given to applicants can vary according to the type of business and required technology for establishing and running the business, 7. The government might set priority sectors/industries for which grants will be given.

11. Internship: Findings (2010)
No significant effect immediately after program end
Large positive effects on program participants on longer term (networking, new skills, initial on-the job training, etc.):
- 25 p.p. higher chances of participants to be currently employed,
- 21 p.p. lower probability of participants to be unemployed at any time after program end/cut-off date
- 23 p.p. higher employment rate of participants based on self-assessment
No significant effect on financial situation or wage

12. Internship: Findings (2012)
Similar findings, long –term effects
No immediate effect: 17 p.p. lower probability of participants to be employed at program end/cut-off
Long-term: 29 p.p. higher probability of participants to be employed currently
More than double higher hourly wages of participants
Small positive difference in inactivity

13. Internship: Suggestions
As the evaluation shows that both programs (2010 and ) are effective, the Ministry may lift the obligation for employers to hire 50% of the interns to make the program more attractive for employers;
We suggest that the government aligns further this program with similar programs (and terminology) of the EU countries;
In line with the experience of EU countries, the length of the internship could be made more flexible (3 to 9 months);
Provide job readiness training before placing young people on internship.

14. TKE: Findings (2010)
Large gains in actual and subjective employment (42 p.p. higher probability of participants to be employed currently)
30 p.p. lower probability of participants to be unemployed at any time after program end/cut-off date
Very large gain in wages for participants (more than three times higher wages)
35 p.p. higher employment of participants based on self-assessment
Self-reported better chances for employment after program end/cut-off

15. TKE: Findings (2012)
Large positive effects on employment: 34 p.p. higher probability of participants to be employed currently
26 p.p. lower probability of participants to be unemployed at any time after program end/cut-off date
Very large gain in wages for participants
28 p.p. lower chances of participants to be inactive today
47 p.p. higher employment of participants based on self-assessment
Reported better chances for employment after program end

16. TKE: Qualitative comparison
Both TKE 2010 and 2012 were found to have led to sizeable gains for the persons selected in the program.
The program showed similar results in the two observed years
Both evaluations suggested the programs have rather longer-term impact
All effects, however, were found stronger in the TKE 2012 program.

17. TKE: Quantitative comparison
No difference between TKE 2010 and TKE 2012
the present design of TKE could be maintained without prejudice that it performs worse than the earlier design
The insignificant results also confirm that the macroeconomic conditions in 2012 were not better than in 2010.
The better results for TKE 2012 obtained in a qualitative fashion may be driven by the fact that as the program matures, the matching of the required skills by the employers and the actual skills offered by the trainees improves in administrative terms, hence leading to better outcomes.

18. TKE: Suggestions
Priority in this program should be given to employers from sectors which are set by the government as priority ones for the future development of the country, or those that are most propulsive sectors and have high growth potential;
An external assessment system can be set up to assess the effectiveness of the training.

19. Wage subsidies: Findings
No significant results/effects of the program
Only difference in employment after program end, but negative for participants
Possible explanations:
- used by employers to overcome short-term needs for workers
- stigma effect

20. Wage subsidies: Suggestions
Introduce training to unemployed prior to program assignment;
Enhance certification program for training offered by employers to increase the long-term value of the training and the experience gained while in program.

21. Training in deficient skills: Findings
No impact of the program on employment outcomes and subjective well-being
Reported better chances for employment after program end
Reported less improvement in the financial situation after program end/cut-off
Results should be treated with caution because of very small number of observations
Continue the work and data collection

22. Training in deficient skills: Suggestions
ESA should improve the monitoring of the quality of training conducted in the training institutions and should require certification of the training (licencing); and/or
ESA needs to improve the methodology behind the skills demand survey, or use other instruments to collect more reliable information on skills in insufficient supply;
ESA could select trainees only from the locations where there is an excess demand (as the mobility within the country is very low), or carefully chose occupations for which it will provide the training, based on availability of training providers and of unemployed persons in the near neighbourhood (and not solely on the skills in deficient supply).

23. IT training: Findings
No impact of the program on employment outcomes and subjective well-being
Results should be treated with caution because of very small number of observations
Continue the work and data collection (most participants could not be reached)

24. IT training: Suggestions
Changes in eligibility and selection (towards unemployed with shorter spell of unemployment, with better initial IT skills and knowledge);
Imposing some requirements to the program participants, obligation for regular update of contact information 5 years after program end, etc.;
Co-financing of the training;
Program design should be changed or monitoring of former participants to the program improved. The ESA should closely follow the participants through years, and develop a longitudinal database.
Otherwise, the program is costly for the society.

25. Towards policy-making
IA can provide much needed evidence
Close cooperation with the LM institutions as to assess the “right” elements
Timing
Ensuring regular evaluations (plan of evaluations)
Readiness of the policymakers to change and improve ALMPs
Improving experience exchange in ALMPs in WB6