1. Assessing the Distributional Impact of Social Programs The World Bank Public Expenditure Analysis and Manage Core Course Presented by: Dominique van de Walle DECRG March 21-24, 2005 Presented to:

2. The World Bank PEAM Core Course 2 The evaluation problem 1.Who gains from the programs? Who uses public services? At what cost? Who benefits from subsidies? Who are the target groups? How should transfers be allocated? 2.How much do they gain? Is there more poverty with or without a policy? How much impact will programs have on poverty?

3. The World Bank PEAM Core Course 3 The evaluation problem Impact is the difference between the outcome indicator with the program and that without it. However, we can never simultaneously observe someone in two different states of nature So, while a post-intervention indicator is observed, its value in the absence of the program — the counterfactual — is not. The essential problem in evaluation is one of missing data on the counterfactual of what would have happened in the absence of the intervention

4. The World Bank PEAM Core Course 4 To measure ‘impacts’ rigorously we need ex- post impact evaluation techniques Econometric: generally need baseline or panel data Experimental: require randomized assignment Often we must instead turn to other “quick & dirty” approaches that examine spending "incidence The most commonly used is Benefit Incidence Analysis (BIA)

5. The World Bank PEAM Core Course 5 What is benefit incidence analysis? Step 1: rank individuals by welfare indicator Access to services Step 2: identify usage/participation Utilization Step 3: attribute "gain" or benefit identified by unit cost of providing service Incidence of spending

6. The World Bank PEAM Core Course 6 STEP 1: Access to infrastructure in rural Vietnam (% rural population with the infrastructure)

7. The World Bank PEAM Core Course 7 Step 2: Participation in public works and a means- tested credit subsidy in Maharashtra, India Consumption expenditure per person

8. The World Bank PEAM Core Course 8 Step 3: A typical example of a benefit incidence analysis:

9. Health spending in Kenya, 1992 0 20 40 60 80 100 020406080100 Cumulative population Cumulative subsidy/income Primary Hospital All health Income

10. The World Bank PEAM Core Course 10 Advantages of traditional benefit incidence analysis… Easy to do and to present (with caveats) Disadvantages and limitations … Strong assumptions Do not explain incidence outcomes No specific policy implications

11. The World Bank PEAM Core Course 11 Traditional benefit incidence analysis may... 1. Wrongly assume that the cost of provision reflects the benefit to user 2. Be sensitive to method of ranking households in the original position spatial prices, comprehensiveness of welfare indicator, demographics

12. The welfare measure matters for primary education in Ghana 0 20 40 60 80 100 020406080100 Cumulative population Cumulative subsidy Per capita expenditure Adult equivalent expenditure

13. How quintiles are defined matters for health in Ghana 0 20 40 60 80 100 020406080100 Cumulative population Cumulative subsidy Household quintiles Population quintiles

14. The World Bank PEAM Core Course 14 Traditional benefit incidence analysis may... 3. Mispecify the counterfactual: BIA ignores behavioral responses ex: figure of transfers in Yemen Conclusions about targeting & incidence depend on how the counterfactual is defined

15. Distribution of public and private transfers in Yemen, 1998, by deciles of per capita expenditures excluding transfers (annual YR per capita)

16. Distribution of public and private transfers in Yemen, 1998, by deciles of per capita expenditures including transfers (annual YR per capita)

17. The World Bank PEAM Core Course 17 Traditional benefit incidence analysis may... 4. Give an incomplete picture of welfare effects how did other dimensions of welfare (eg health literacy, nutrition) improve as a result of subsidies? 5. Be unable to assess some important public goods and services eg safe water, sanitation, vector control, physical infrastructure

18. The World Bank PEAM Core Course 18 Traditional benefit incidence analysis may... 6.Ignore general equilibrium & indirect effects on poor eg indirect benefits from tertiary education 7. Confound average and marginal incidence Distribution of gains

19. The World Bank PEAM Core Course 19 7. Average versus marginal incidence Standard benefit incidence estimates the distribution of average incidence at one point in time This can be deceptive about how changes in public spending will be distributed Marginal incidence is an example of a behavioral incidence analysis where one measures the incidence of actual increases or cuts in program spending

20. The World Bank PEAM Core Course 20 How will gains from social program expansion be distributed across groups? Non-poor often capture benefits of (even targeted) social programs information, incentive & political problems make perfect targeting hard But, program capture by non-poor can differ according to how costs & benefits of participation vary with program scale eg fees, opportunity costs of time, transport costs etc

21. The World Bank PEAM Core Course 21 How will gains from social program expansion be distributed across groups? Model 1: Early capture by non-poor: net gains to non-poor are positive initially, but fall with program expansion Model 2: Late capture by non-poor: cost initially too high for the non-poor, but net gains rise over time Average participation rates may be deceptive for inferring how gains & losses from program expansion or contraction will be distributed.

22. The World Bank PEAM Core Course 22 One way to identify marginal incidence is to compare incidence across geographic areas with different program sizes Marginal odds of participation (MOP) = increment to group-specific participation rate with a change in overall participation The income group specific MOP is estimated by regressing income group specific participation rate across regions on the average rate for the region. MOP shows incidence of a change in spending

23. The World Bank PEAM Core Course 23 Average and Marginal Odds of Primary School enrollment, India 1993-94 Note: odds of enrollment = ratio of quintile-specific enrollment rate to the mean rate.

24. The World Bank PEAM Core Course 24 In conclusion Extreme care is needed when interpreting average incidence and traditional benefit incidence analysis Beware of reform recommendations based solely on BIA and concentration curves as conventionally calculated.

25. The World Bank PEAM Core Course 25 Q & A