1. Social Assistance Pilots Program SA Pilots Seminar Hybrid Means Testing (HMT) Model Development Roman Semko CASE Ukraine March, 2010

2. 1.Introduction to modeling 2.Data analysis 3.Methods for estimation 4.Simulations 5.Income from assets (agriculture) 6.Double-blind experiment results 7.Model comparisons and conclusions Content Content 2

3. Concept Concept 3 The World Bank has developed a methodology for income estimation which is based on regression analysis – HYBRID MEANS TESTING (HMT)The World Bank has developed a methodology for income estimation which is based on regression analysis – HYBRID MEANS TESTING (HMT) Under HTM method, eligibility to the SA program is assessed based on the households income modelingUnder HTM method, eligibility to the SA program is assessed based on the households income modeling Total income is divided into two parts: easy to verify (e.g., pension, stipend) and hard to verify (e.g., dividends, shadow wage)Total income is divided into two parts: easy to verify (e.g., pension, stipend) and hard to verify (e.g., dividends, shadow wage) The final goal is to estimate hard to verify share of the income based on a set of variables, which can be accurately measured and reflect the hard to verify incomeThe final goal is to estimate hard to verify share of the income based on a set of variables, which can be accurately measured and reflect the hard to verify income Hard to verify income is divided into income which is not generated by long-term assets (estimated by regression model) and income from assets (estimated by formulas)Hard to verify income is divided into income which is not generated by long-term assets (estimated by regression model) and income from assets (estimated by formulas)

4. The main goal of the model is to predict most precisely total family income The main goal of the model is to predict most precisely total family income Source: Finance Ministry of Ukraine Data and Knowledge Methods Equation which estimates applicant’s income based on the available information: Y = β 1 *X 1 + β 2 *X 2 + β 3 *X 3 + … YX 1, X 2, X 3, …  total income  hard to verify  family structure  type and sector of employment  education  region  other Model Criteria 1.Theoretical validity 2.Simplicity 3.Goodness of fit 4.Significance of explanatory variables Application and Simulation 4

5. Pilots datasetHBS 2008 Good model should use all available relevant information for income prediction Good model should use all available relevant information for income prediction 5 10,622 observations of households with total income > 3,000 observations of families with declared income Cannot be used separately for model estimation since total income is not available A lot of information could/should be used to guarantee acceptable level of precision Declared income (DI) is an important indicator in total income (TI) assessment MATCHING CharacteristicsTICharacteristicsTIDICharacteristicsDI

6. Pilots dataset HBS 2008 Observations are matched in a way to guarantee the highest similarity between them Observations are matched in a way to guarantee the highest similarity between them 6 Procedure 1. Form groups based on the follow-ing variables: type of settlement, type of assistance, household’s size, # of children, working persons, pensioners, sex of the single-heads household 2. Match each observations from HBS to the observation from pilots dataset from the same groups based on the similar characteristics: age of the head, education of the head, etc. using Euclidean distance function 3. Each observation from pilots dataset is used for matching no more than 2 times 4. Aggregate the groups if there are no good candidate for HBS observation from corresponding group from pilots dataset and match again Observation 1 Observation … Observation K 1 Group 1 Observation 1 Observation … Observation K … Group … Observation 1 Observation … Observation K N Group N Observation 1 Observation … Observation L 1 Group 1 Observation 1 Observation … Observation L … Group … Observation 1 Observation … Observation L N Group N

7. Total vs. declared income comparison (without SA) Data comparison: a main difference between HBS and pilots applicants occurs in their incomes, while most of other characteristics are similar Data comparison: a main difference between HBS and pilots applicants occurs in their incomes, while most of other characteristics are similar 7

8. For some regions average income in HBS significantly differs from the Personal Disposable Income (PDI) Statistics For some regions average income in HBS significantly differs from the Personal Disposable Income (PDI) Statistics 8 Statistics, PDI HBS Lviv Lutsk Rivne Ternopil Ivano- Frankivsk Uzhgorod Chernivtsi Vinnytsa Zhytomyr Chernigiv Sumy Kharkiv Poltava Cherkasy Kirovograd Odesa Kherson Mykolayiv Zaporizhzhya Dnipropetrovsk Lugansk Donetsk Simferopol Kyiv Khmelnytsky Differences in income without SA per capita compared to Chernivtsi region, UAH – >200 – 100-200 – <100 Lviv Lutsk Rivne Ternopil Ivano- Frankivsk Uzhgorod Chernivtsi Vinnytsa Zhytomyr Chernigiv Sumy Kharkiv Poltava Cherkasy Kirovograd Odesa Kherson Mykolayiv Zaporizhzhya Dnipropetrovsk Lugansk Donetsk Simferopol Kyiv Khmelnytsky

9. Bayesian econometrics allows combining data with aggregated publications of regional PDI Bayesian econometrics allows combining data with aggregated publications of regional PDI 9 Standard estimation Bayesian estimation Calibration Researcher artificially determines the model coefficient(s), e.g., if regional macrodata say that income in Kyiv city is 1108 UAH higher than in AR of Crimea, than it is assumed that for Kyiv city applicants income is 1108 UAH higher than for AR of Crimea applicants, other things equal Coefficients are determined based on the collected observations using standard regression tools (classical econometrics) Combines both approaches. Estimated coefficient lies between calibrated and estimated in a standard way Does not lead to significant changes within regions but for regions across Ukraine changes are significant: average predicted income for regions has changes

10. Linear model is the most simple Linear model is the most simple Linear Description R2R2 Predictions 58 % (large cities – 65%, small cities – 63%, villages – 48%) Linear relation between income and family characteristics Dependent variable is under the logarithm (log-linear) Independent variables (IVs) include easy to verify income Other IVs are: number of children, of working persons, of the elderly, type and sector of employments of household heads, education level – declared income – predicted income Concept: the more income the applicant declares, the lower the additional predicted income is – a sort of a “zero sum game” 10

11. Nonlinear model is performing well when income differences are high – for the whole HBS sample Nonlinear model is performing well when income differences are high – for the whole HBS sample NonLinear Description R2R2 Predictions R 2 -square is not bounded in [0%,100%] region Nonlinear relation between income and family characteristics. The form of relation: cubic or quadratic – since total income sorted in ascending order increases as a polynomial of 2 nd or 3 rd order Dependent variable is under the logarithm (log-linear) Independent variables are as in the linear model – declared income – predicted income Concept: as for the linear model 11

12. Two-step model is effective when there is a large number of families with zero and nonzero hard to verify incomes Two-step model is effective when there is a large number of families with zero and nonzero hard to verify incomes Two-stage Description R2R2 Predictions 47 % (no division by cities) At first stage probability that family has shadow income is estimated and then linear relations between income and family characteristics with a hazard of having shadow income is used for estimation Dependent variable is under the logarithm (log-linear) and does not include salary – declared income – predicted income Concept: Stable additional income is added to the declared – “the game with constant markup”. 12

13. 1. DEPENDENT VARIABLE Informal (shadow) salary was incorporated into the dependent variable (hard to verify income) since it is not easy to verify income Each model needs a set of adjustments in order to become fully useful Each model needs a set of adjustments in order to become fully useful 13 Adjustments 3. TIME INCONSISTENCIES In order to compare incomes across different time period, average growth rates of PDI and its elements were used for time adjustment 2. EXPLANATORY VARIABLES (EVs) Some EVs which can be used for predictions are hard to verify, e.g., number of mobile phones cannot be accurately measured 4. FAMILY HEADS The definitions of family heads are standar- dized: male co-head and female co-head are used instead of voluntary definitions Prediction does not change significantly unless dependent variable is redefined. If the dependent variable is redefined, additional predicted income becomes more stable and decreases with the increase of declared income at a lower rate Description

14. Average predicted income exceeds declared by 26% Average predicted income exceeds declared by 26% 14 Declared vs. Predicted income (by models)

15. 27% families will be excluded from the SA programs 27% families will be excluded from the SA programs 15 Number of beneficiaries (hypothetical scenario)

16. Average assistance will drop significantly, except for low income and fuel subsidies Average assistance will drop significantly, except for low income and fuel subsidies 16 Average assistance (hypothetical scenario)

17. Total budget for SA expenditures will decrease by 27% Total budget for SA expenditures will decrease by 27% 17 Total expenditures on SA (hypothetical scenario)

18. New approachCurrent situation Income from agriculture assets is calculated based on the developed normatives Income from agriculture assets is calculated based on the developed normatives 18 Agriculture income is calculated as income per hectar Normatives are not unified across regions Income calculation per hectar and per each animal Differentiation between cities and villages Normatives are unified since they are based on the same methodology and data Calculation procedure Information, certified by the village/city council Is not applied to families with disable persons or elderly (>70) If applicant lives closer than 10 km to the city – apply city normatives Income from land is a product of land area and normatives Income from payi is calculated sepatately Income from lifestock is the product of number of livestock heads times the normative Average predicted income exceed declared by 28%

19. New approach Current situation Example of income calculation from agriculture Example of income calculation from agriculture 19 NORMATIVELAND AREANORMATIVEANIMALSAGROINCOME CROPSANIMALS Only farm- stead area of 0.56 hectars, located in village (Donetsk region) 127.62 per hectar per month Possess one cow and 10 chickens Only related through the hayfields and pasturage Only farm- stead area of 0.56 hectars, located in village (Donetsk region) 412.44 per hectar per month Possess one cow and 10 chickens (the same) 270.83 for cow, and 4.48 for one chicken + 63.81 UAH per month + 521.85 UAH per month

20. Declared Income = 211 UAH Double-blind experiment: case study Double-blind experiment: case study 20 Family description Father: unemployed and not registered in employment center Mother: housewife Age: <18Age: <3 Commission case and home visit DENIAL Model result LESS than Eligibility threshold = 255 UAH Model prediction = 308 UAH immediate decision – risky family, need home visit GRANT? WHILE MORE than Eligibility threshold = 255 UAH DENIAL

21. Cases of SA denials through commission, based on home inspections Cases of SA denials through commission, based on home inspections 21 Type of assistance Declared income Predicted income Absolute difference Relative difference UAH % Low income91331241320 Low income2113089746 Low income2643448030 Child care101331231238 Child care12127114931 Child care2092867737 Child care2503277731 Child care26036510540 Child care2733305621 Single mothers068 - Single mothers2222916931

22. Predicted income helps to select families for home inspection Predicted income helps to select families for home inspection 22 Families selected for inspection Comments Each family has a chance to be selected for home inspection The probability of selections increases as the predicted income is significantly different from declared in absolute and relative terms

23. Model comparison Model comparison 23 Characte- ristics MODEL Linear (no matching) NonlinearTwo-step model Linear (matching) Linear (Bayesian + matching or not) Theoretical background Weak since linear relations are rare in nature Stronger since takes into account nonlinearities Strong if selection bias is expected to be Same as linear (no matching) SimplicityVery simpleSimpleSlightly complex Simple in estimation but complex in data matching Very complex in estimation R-squareHigh- - Information as an input Only HBS HBS and pilots dataset Very effective use of information Influence on applicant High MediumHigh

24. Conclusions Conclusions 24 Income estimates generated by the models significantly differ from the incomes declared by the SA applicants Further empirical tests with the models are needed Initially model results should be used only as an advice rather than a criterion for granting SA benefits The models may be used as an instrument for selecting families for home inspections 4 3 2 1