1. L. Elia, A. Morescalchi, G. Santangelo
Evaluation of an ESF funded training program to firms: The Latvian case
L. Elia, A. Morescalchi, G. Santangelo
Centre for Research on Impact Evaluation (CRIE)
Joint Research Centre, European Commission
Andrea Morescalchi
Ministry of Finance, Riga (LV)
10-11 March 2015

2. CRIE: What is it & Who is in it?
Centre for Research on Impact Evaluation Joint DG EMPL-DG JRC initiative
Established in June 2013
Support to MS and DG EMPL to set up the necessary arrangements for
carrying out Counterfactual Impact Evaluations (CIE) of ESF
funded interventions
Who?
Researchers with a background in Economics and Statistics

3. at DG JRC  in its Ispra (Italy) site 
CRIE: Where?
at DG JRC  in its Ispra (Italy) site 

4. DG EMPL’s initiative to help MS in this shift toward impact evaluation
How was CRIE born?
Difficult economic times
Even more pressing need for the EU to demonstrate the achievements of ESF and other financial instruments
programming period:
 shift toward impact evaluation
DG EMPL’s initiative to help MS in this shift toward impact evaluation
... CRIE in collaboration with DG JRC

5. 1. The intervention

6. The intervention – 1st call
Type: training to employees of firms. The type, duration and number of trainings can vary over firms and employees within firm.
Motivation: improve the performance and competitiveness of firms by enhancing the skills and the productivity of their employees.  
Eligibility: associations formed by at least 5 firms belonging to the same industrial sector can apply.  
Duration: March April 2010.
Participants: 14 projects selected for a total of 111 firms

7. The intervention – 2nd call
Duration: January 2011 – July 2015.
Very similar to 1st call except:
total turnover of project co-applicants should exceed million Euro in the last year 
the applicant association should have been at least 3 years in operation
Larger group of recipients (>1,000 firms)

8. 2. Related literature on training to firms

9. Human capital theory Returns to on-the-job training estimated by:
individual productivity measures
wages: proxy for individual productivity
employment prospects:
results of enhancing individuals’ skills and competencies
(Latvia: Dmitrijeva and Hazans, 2007)
overall firm level outcomes (non-CIE methods)
value added or sales (absolute or per-capita value):
measures of firm performance
Training characteristics to be taken into account:
Type
Hours
Duration
Costs

10. Explanatory variables
Firm
Capital
Labor
R&D
Sector of activity
Size
Location
Workforce (shares)
Occupation
Education
Experience
Age
Gender
Worker turnover

11. 3. The data

12. Data description Data sources:
project database from the Investment and Development Agency of Latvia (IDAL): # of employees receiving training;
business data collected from the State Revenue Service;
Period: Yearly firm level data available for
Variables: Firms’ age, # of employees, location, NACE code, (after-tax) turnover, production costs, profit, salaries, assets, capital investments, fixed assets, long-term intangible investments

13. Data description – 4 main firms groups
Treated in 1st call: 111
Treated in 2nd call: 1,118
Never treated: 30,557 (nearly 40% of the country total),
Treated in both calls: 38, not used in the analysis
Sample adjustments:
missing data and outliers removed from the sample: group (1) drops to 66
firms starting 2nd call before December 2010 are removed: group (2) drops to 391!

14. Data description – sample statistics

15. Data description – profit per employee

16. Data description – turnover per employee

17. 4. Methodology

18. DiD combined with PSM
Technique: combination of Difference-in-differences (DiD) with Propensity Score Matching (PSM).
PSM: used to select a control group of firms similar to treated firms in terms of observed characteristics.
DiD: applied on the sample of treated and untreated peers.

19. Propensity score matching
Propensity score matching (PSM) is used for selecting control units from two comparison groups in 2008
Variables used for matching: capital investment, long-term intangible investments, production costs, pre-program profit and turnover (all normalized by the number of employees), age, number of employees, NACE sector of activity and geographical location
Four algorithms are employed: one-nearest neighbor, with and without replacement, three-nearest neighbors and kernel matching

20. Difference-in-differences (DiD)
DiD regression model estimated after matching untreated to treated:
𝑦 𝑖𝑡 = 𝛼 𝑖 + 𝜆 𝑡 𝑡=1 𝑇 𝑦𝑒𝑎𝑟 𝑡 +𝝆 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑖 ∙ 𝑑 𝑡 + 𝛽 ′ 𝑥 𝑖𝑡 + 𝑢 𝑖𝑡
𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑖 is the share of employees in training
𝑑 𝑡 =1 if 𝑡≥2008
𝑡=1 𝑇 𝑦𝑒𝑎𝑟 𝑡 are time dummies
𝑥 𝑖𝑡 are observed firms’ characteristics
𝛼 𝑖 captures time-invariant unobserved firms’ attributes
One model is estimated for each of the two control groups

21. 5. Results

22. Results for profit per employee
Short-term impact

23. Results for (log) turnover per employee
Short-term impact

24. 6. Limitations

25. Results to be taken with extreme caution…
Information on the kind, duration and number of training courses taken by employees is not available
exact start and end date of treatment not available
strong selection into treatment
association identifier not present in the data
information not available at the individual level
size of the treatment group is very small
only 1 year available before the intervention

26. 7. Concluding Remarks

27. Insights and recommendations
The limitations encountered in the present evaluation make it difficult to discuss the replicability and the scaling up of the intervention under scrutiny.
Future availability of more detailed data and information will be critical to produce more reliable and robust impact evaluations