1. agriregionieuropa Evaluating the CAP Reform as a multiple treatment effect Evidence from Italian farms Roberto Esposti Department of Economics, Università Politecnica delle Marche 122 nd European Association of Agricultural Economists Seminar Evidence-Based Agricultural and Rural Policy Making Methodological and Empirical Challenges of Policy Evaluation February 17 th – 18 th, 2011, Ancona (Italy) associazioneAlessandroBartola studi e ricerche di economia e di politica agraria Centro Studi Sulle Politiche Economiche, Rurali e Ambientali Università Politecnica delle Marche

2. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Why don’t use powerful treatment-effect econometrics to asses 2003 CAP reform? – Now we have micro-data! But: 1.CAP is a multipurpose and multioutcome policy 2.CAP is a multitreatment policy 3.We still have observational (non-experimental) data Objective – Estimate the ATT of 2003 Pillar I Reform -The treatment: the 2003 Pillar I Reform -The effect: market orientation (≠ production choices) The idea and the objective

3. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) The sample  A balanced panel (constant sample) of 6543 farms FADN farms Random sampled Representative of professional farms Representative of different geographical conditions Obs. over years 2003- 2007 (pre and post- reform). But: Strongly heterogeneous

4. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  The 2003 CAP reform had several objectives – Which of them is actually prevalent? – Which of them do we want to evaluate/assess?  Decoupling: the key of the reform – Market orientation:  Let farmers choose what (and if) to produce  Let farmers achieve an higher allocative efficiency 1. CAP reform as a multipurpose policy The main objective of the Reform was to induce farmers to change their output vector

5. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  How do we measure if and to what extent farms changed their output vector? – Two different types of outcome: In a short-run perspective: change in the composition of output. Measures of distance between pre (A) and post (B) In a long-run perspective: investment decisions 1. CAP reform as a multioutcome policy Output-distance index Investment rate

6. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  The CAP is a multi-measure policy: 2. CAP reform as a multitreatment policy (1) 2003 CMOs RDP Total number of treatments: 2003 = 2,6 per farm on avg.; 12600 € per farm on avg. 2007 = 3,0 per farm on avg.; 14700 € per farm on avg. Distribution of support across measures (>1%):

7. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Pillar I reform can not be assessed without taking into account that: – Many farms receive the Pillar II treatment, too – This treatment affects the same outcomes Change SR and (above all) LR production decisions → This treatment interacts with Pillar I Reform: multiple interacting treatments Which is the effect of Pillar I Reform? It depends on whether the farms also receive Pillar II measures There are multiple treatment groups There are multiple counterfactuals (control groups) 2. CAP reform as a multitreatment policy (2) 2003

8. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Three groups: – the control group (C farms): no CAP support - 938 farms – the “Pillar I regime change” treatment group (T I farms): shift in Pillar I support, no Pillar II support - 3775 farms – the “Pillar I regime change with Pillar II” treatment group (T II farms): shift in Pillar I support at least one Pillar II payment - 1829 farms  Three treatments: – Treat. 0: T I farms vs. C farms – Treat. I: T II farms vs. T I farms – Treat. II: T II farms vs. C farms 3. From observational to quasi- experimental data

9. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Selection on (un)observables: – There can be variables that affects both the treatment assignment and the outcome. For instance: production specialization (Type of Farm) – Selection on (un)observables may occur but can not be easily detected: A.Strong heterogeneity across the groups B.Strong heterogeneity within the groups How can we isolate the effect of the treatment from the effect of these (un)observables? – Suppose they are observable...: Unconfoundness 3. Selection bias

10. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Identifying the ATT (es. Treat. 0): ATT: identification and estimation Just the difference between the conditional expected value of the treated and the control groups because: Selection bias is eliminated by unconfoundness: all confounding variables are conditioning variables  Estimating the ATT – 3 steps: 1.Match control and treated groups on the basis of X 2.Calculate ATT on each match 3.Average the ATTs

11. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  An easier matching – The matching is based on a scalar p (X) rather than on a vector X of covariates – p (X) synthesizes the covariates that really count in treatment assignment – Pair-wise (or group-wise) matching can be easily made on p (X) – Unconfoundness can be controlled (balancing  p (X) is simply based on estimation of a PSE (es. binomial probit) Propensity Score Matching

12. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Four alternative ways of finding matches: – Stratification Matching: comparisons between blocks based on p (X) of controls and treated units – Radius Matching: any treated unit is compared to controls within a radius of p (X) – Nearest Neighbour Matching: one-by-one comparison with the control with the closest p (X) – Kernel Matching: any treated unit is compared with all control units each weighted by the inverse of its distance from the p (X) Alternative matching

13. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy) Results: propensity to be treated  What affects treatment assignment: – Specialization (TF)! – Size and age: smaller+older farms exclusively Pillar I treat. – Labour-intensive lowland farms: no policy – Capital-intensive hill-mont. farms: both policies Balancing property satisfied

14. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Treatment 0: only Pillar I reform vs. no policy – Output-distance index: ↑ >50% – Investments: weakly significant  Treatment I: only Pillar I vs. both Pillars – Output-distance index: ~ half effect – Investments: ↑ > 30 %  Treatment II: only Pillar I reform vs. no policy – Output-distance index: ↑ > 75% – Investments: ↑ >60% Results: the estimated ATT

15. agriregionieuropa 122 nd EAAE Seminar, February 17 th – 18 th, 2011, Ancona (Italy)  Did the Pillar I reform reorient production decisions? YES  Short-run vs. Long-run production decisions – Pillar I reform affects SR decisions – Impact on LR (inv.) decisions is questionable – Major impact on LR decisions from Pillar II  Interaction between treatments – The LR impact comes from the complementarity of the two Pillars  Robustness and sensitivity analysis: to do – Why do ≠ matching methods differ (not so much)? – Tradeoff between bias and variance (precision) ► Bounds and sensitivity analysis Conclusions