1. INVESTMENT CLIMATEDEVELOPMENT IMPACT EVALUATION INITIATIVE Innovations in Investment Climate Reforms: An Impact Evaluation Workshop Takeaways and Next Steps

2.  Understanding impact evaluation  Impact evaluation (IE) vs. Monitoring  IE as an operational tool for reform design and results attribution  Overview of methods  Evaluations under the IC impact program  Research questions  Key methodological points to keep in mind  Next steps Outline

3.  Monitoring  Offers a way to track progress over time and whether projects are moving in the “right direction”  Tracks only those that are receiving the intervention  Does not measure if our intervention caused certain outcomes  Impact evaluation  Identifies the underlying mechanism, tells us why things happen or how to make them happen  Tracks both, those receiving the intervention (treatment group) and a group with similar characteristics as the treatment group that is not receiving the intervention (control group)  Measures the outcomes caused by our intervention IE vs. Monitoring

4.  IE can help answer the “big” questions: e.g. what is the impact of the business registration reform on job creation? Does tax reform promote formal firm creation? How do risk-focused inspections impact business growth and private investment?  BUT, is also critically important as an operational tool to help test variations in project design to maximize impact: e.g. which value added services are most effective in maximizing the impact of the registration reform on job creation? Do thresholds of eligibility for tax reforms de-courage firms to declare a higher fraction of their income? How does information on the probability of being inspected influence compliance? IE as an operational tool

5. IE as an operational tool: Example for business registration Test…a question…By comparing different options Variation in intervention What is the price elasticity of demand for registration fees? Offer subsidies: $100; $50; $10 to different groups Complementary interventions Can financial training on banking products improve benefits of formalization? Pilot a training program (or other value added service) InformationWhat is the most effective way to communicate the new reform? Test phone calls, flyers, public events, radio adverts on different groups Firm behaviorDoes framing influence firm perceptions? Highlight the benefits of registering to one group and the threats/risks of not registering to another Supply side incentives Can non-financial staff recognition improve processing times? Offer non-financial vs. financial incentives to public service staff.

6. IE as an operational tool: for better informed decisions  Improve the design and implementation of CIC projects Impact assessments will measure the overall impacts of interventions, test their underlying mechanisms, combinations with other interventions and sequencing.  Improve strategic decision making and resource allocation Findings on impacts will improve strategic choices on the project portfolio, and support our evidence-based approach to policy making through a better understanding of what works, what doesn’t, where, why and for how long.  Share with client countries and development partners Increased knowledge on various impacts will support the decision-making process of our donors’ related to prioritization of reform topics and funding.

7. Overview of methods MethodDescriptionPrerequisitesPossible application RandomizationRandom assignment of individuals and firms to treatment and control groups. Ensures the groups are identical (on average) on observed and unobserved characteristics ( considered the ‘gold standard’ sin IE: simple, precise, cheap) Sufficient sample sizeMatching grants, information campaigns, training, value added services for businesses registration, inspections etc Regression Discontinuity Design (RDD) RDD works as long as there is a clear cut-off point for some index/variable that can map people/businesses into being closer or further from being given the treatment. Ranking and distribution can be used interchangeably (the idea is that businesses can be ordered in some way and there are no jumps in the distribution around the cut-off). Intervention has a clear cut-off for eligibility, e.g. length of business establishment for qualifying credit, distance from a geographic boundary if the intervention is allocated geographically, or proximity to a cutoff eligibility score such as turnover level for tax regime or application rating for a matching grant. This requires a large number of applicants falling close to the cut-off region. Tax regime change for income ranges, trade laws/logistics for particular quantities/ goods etc. Encouragement design. Used when the reform/program is accessible by all at once. A randomly selected group is encouraged to participate, making them more likely to take up the program (e.g. register their business) and is compared to a ‘non- encouraged’ control group. Encouragement or information campaign is (i) applied randomly to potential recipients and (ii) has a strong effect in improving awareness and take up of the program In cases where take up for the intervention/reform is low, but can be improved through targeted marketing. E.g business registration reform: appropriate as long as the encouragement does not change outcomes in any way other than through making the firm more likely to apply for the intervention (for instance, offering free business banking would not be a good instrument for encouragement - it would increase take up but the encouragement itself would also have an influence on outcomes of interest beyond through the effect of increasing take up. This is why info campaigns are more reasonable options).

8. Evaluations under the IC impact program  Benin Registration “Entrepreneurial” Status  Nepal Registration Automation  Bangladesh Licenses  Kenya Health Safety Inspections  Tajikistan Risk Based Tax Audits  South East Europe Trade Logistics  Kosovo Investment Policy Incentives  Romania Insolvency Reform  ECA Food Safety Standards

9. Research questions BeninWhat package of incentives (information, finance, tax) has the highest impact on formalization rates, behavior and performance of newly formalized firms that are offered ‘Entreprenant status’? NepalWhat is the effect of simplification, automated registration and e-services on registration rates (in excluded regions) and job creation? What value added services could improve the impact of formalization on business growth? BangladeshWhat is the impact of license reform on small firms? What are the triggers and constraints of firm formalization? KenyaWhat is the effect of risk-based support for the health sector on the quality of service of health facilities? What is the optimal frequency? What is the effect of distance support (and a SWAT team) in improving compliance and client satisfaction? TajikistanWhat is the impact of the tax simplification reform, in particular of risk-based audits, on revenue collection? Does risk-based audit reduce corruption? South-East Europe Trade Logistics (Albania, Serbia, Montenegro) How does the introduction of a scanner and of risk based border clearance (i.e. green channel) impact border crossing time, trade flows at border posts (by type of product) and trade characteristics (number of firms, average price, trade patterns)? KosovoWhat is the effect of sector-specific grants on FDI, employment and innovation? How do different targeting schemes and implementation combinations effect these benefits? RomaniaDoes increased awareness about out-of-court debt restructuring practices improve loan repayment rates and new loans granted to eligible debtors? ECA Food Safety Standards (Moldova, Bosnia & Herzegovina) What is the impact of the Free Trade Agreement with EU on Moldovan agricultural export and import volumes, destination and portfolio? What is the role of food safety work in these results and what are the most effective FS dissemination/ training approaches to ensure compliance?

10.  Intervention take up  Multiple treatments  Sub-group analysis  High-frequency data  Sampling  “General equilibrium” effect and spillovers  Realistic timelines Key methodological points to keep in mind

11. Intervention take up  Low take-up may reduce chances of measuring impact. Need to think about innovative techniques to ensure high take up (using behavioral lessons). Nepal/Benin/Bangladesh  Measuring the effect of formalization on jobs requires a large increase in registration probabilities. Worthwhile having a “treatment arm” that is designed specifically to increase take up dramatically with the objective of measuring the effects of formalization. For example, a handholding activity (basically doing everything for firms to ensure formalization, following the Malawi IE example) can be used to ensure high registration take up and to test how the act of formalization impacts business growth and jobs..  Specify the strategy on how to target informal businesses Romania  Relies on encouragement design. Measure if take up differs across regions to assess if some firms are less accessible to the training location. Highly indebted people are also less likely to take calls or open letters.  Intervention might vary depending on instructors and influence results (will the training course be always the same?). Also, make sure to clarify eligibility criteria.

12. Multiple treatments  Multiple treatments compare effects across different interventions and their combinations. The more comparisons you make, the larger the sample you need to be “confident”. Different treatments also affect various populations, and results should be interpreted accordingly. Benin  The IE goes beyond usual business registration IEs, which focus on the effects of registration simplification (and possibly information campaigns) on registration rates, by studying the effects of different incentives packages (registration+information,+microloan,+tax, and combinations thereof) on business performance. Each package entails different take up rates and populations. The team should make sure to interpret impacts based on the characteristics of populations receiving specific treatments. Nepal  What are the different ICT components to be tested as a part of the new e-engineering and automating registration process? Consider adding several value added services (VAS), in addition to a business directory, to test the economic benefits of such registration? Benin/Kenya  Several treatment arms will require a large sample. Bangladesh/Tajikistan  Consider other useful VAS for businesses/taxpayers (Tajikistan: consider measuring corruption). Kosovo  Not feasible to test all types of incentive schemes, but could assess some variations.

13. Sub-group analysis  Important, but requires bigger samples. Need to stratify sample at start: male vs. female entrepreneurs, small vs. bigger businesses/health facilities, different business sectors or region etc., depending on the theory of change and what we want to test. Benin  Targeting different segments of informal businesses is likely to matter a lot (i.e. who gets chosen to be an “entreprenant”). The team could be more open in the selection criteria at first, then analyze the impact for different subgroups to help refine and define the final targeted sub-group. Nepal/Bangladesh  Nepal proposes rural/urban sub-groups to test how online registration system overcomes geographical barriers. Both IEs could consider including business sector and gender criteria. Tajikistan  What are the indicators to identify “risky” businesses for auditing? Consider sub-sample analysis for some of these selected measures to improve understanding of which types of businesses are most likely to be influenced by RBA. SEE Trade  Which sectors are likely to benefit most from the risk-based inspections? Important to consider perishable vs. non-perishable products as well as products with volatile vs. non-volatile market prices.

14. High-frequency data  Should be exploited whenever available: they can strengthen estimates of important indicators, complement other data sources and are less costly then collected data. They are usually available when administrative data of interest is regularly updated (trade data, daily business registration rates, bank data on loan repayments/default rates, judicial court case data, etc.) Benin/Bangladesh  Consider exploring high-frequency data (especially Bangladesh, given the regional/municipality based roll-out) and obtain historical and future registration rates for treatment and control in the lowest aggregation possible (e. g. daily). Kenya  Consider using high-frequency administrative data on health facility performance and satisfaction surveys at the facilities to reduce the cost burden for data collection. SEE Trade/ECA Food safety  Will need to rely on high-frequency data and trend breaks at the times of intervention effectiveness. A potential challenge is that the time between project effectiveness and awareness by users can be fuzzy. Consider ways to sharpen this by providing an information campaign if the intervention is still to take place. Alternatively try to exploit natural variations in the types of businesses that are most likely to become aware of the intervention most immediately and consider sub-sample analysis on these groups. ECA Food safety  Expand on the discontinuity of the FTA date. Check historical data on whether trends change before and after discontinuity (due to anticipation) or whether these are stable.

15. Sampling  A sampling frame to choose survey participants needs to be carefully designed to obtain credible results. Also, when determining the sample size, teams should consider several factors, but mainly outcomes of interest, the expected change in the outcomes and expected take up rates. Benin  Relies on partner institutions. The risk is that partner institution info may vary greatly in quality. Consider doing an initial screening of lists through direct contact to ensure quality and remove ineligible or closed businesses. Also, consider using tax and other official administrative data to identify informal entrepreneurs. Nepal  Consider using tax data to identify informal businesses. This will be a specific sub-sample of informal businesses; alternatively conduct HH-level listing in survey areas to identify businesses and compile frame. This will only consider business formalization rather than business creation since firms that do not exist cannot be listed. SEE Trade  Matching would require a detailed sampling frame of businesses. Is this available? Alternatively, phased approach at ports. Kosovo  Can we get a big enough sample? Investment incentives may suffer especially from low take up rates, severely limiting the scope for evaluation. If using an encouragement design, think about different ways to derive a sampling frame – for example by consulting the expected outcomes of different intervention variations with local government agencies such as Chambers of Commerce.  Specify the unit of randomization and the overall identification strategy.

16. “General equilibrium” effect and spillovers  Looking at the bigger picture of the project, and its effects, is important. Measuring only the impact on the target groups may miss important spillovers and complement/substitute effects of the intervention. Kenya/Tajikistan  Inspection rates are likely to influence behavior in other health clinics and firms. Varying the “intensity” of inspection by changing the rate of inspection for neighboring centers may help to measure this, but sample size may be a challenge. SEE Trade  How does the change in border processes influence decisions of importers/exporters to move to other border crossings? Will be important to capture regional trade as a whole to make sure any impact being observed is not just a “substitution effect”.

17. Realistic timelines  Align IE implementation with the intervention timeline (ex-ante IEs are usually better as they allow for tailored data collection).  Identify possible implementation bottlenecks, which might have time and cost implications.  Applies to all IEs.

18. Next steps  Evaluation teams  Finalize concept note ( ✔ ) and start planning for methodology note.  CIC impact team  Acts as the CIC focal point and main interface for IE teams (see Annex for the list of CIC focal points and researchers).  Provide technical support and financial assistance to evaluations in the program to advance further on their design and implementation.  Promote partnership with DIME and external experts that provide feedback to the teams and ensure implementation of the most rigorous methodologies that are feasible.  Engage in a series of follow up activities to increase understanding of most appropriate methods to use when impact evaluation are not feasible and how to use findings from one country in another.  Share knowledge on CIC impact evaluation, strengthen the community of practice, and use website as platform for information, tools, and interaction.

19. Milestones MilestoneDeadlineStatus Preparation of concept noteJanuary 15 th, 2013 ✔ Peer reviewed methodologyApril 15 th, 2013 Data collection plan (outputs: TORs, final questionnaire) August 31 st, 2013* First data analysis (outputs: technical note, presentation, and data files) December 31 st, 2013* Implementation of intervention aligned to evaluation Follow up data collection (outputs: TORs, final questionnaire) Final report and policy note (outputs: technical note, policy note, data files, presentation) June 30 th, 2015* * Indicative deadline, may vary by IE

20. Annex: CIC focal points and researchers IE TeamCIC Focal PointResearcher Benin Registration “Entrepreneurial” Status Massimiliano SantiniDavid McKenzie Zoubir Benhamouche Bangladesh Licenses (BICF Program/Sustainability) Syed A. Mahmood Jana Krajcovicova Tbc. Nepal Registration AutomationAna GoicoecheaAidan Coville Felipe Dunsch Romania Insolvency ReformMassimiliano SantiniLeora Klapper Kenya Health Safety InspectionsAna GoicoecheaTomomi Tanaka Tajikistan Risk Based Tax AuditsPenelope Demetra FidasDina Pomeranz South East Europe Trade LogisticsAlejandra MendozaRussell Hillberry Kosovo Investment Policy IncentivesJana KrajcovicovaTbc. Food Safety StandardsAlejandra MendozaTom Reardon Joe Swinnen Georgia Tax ReformMassimiliano SantiniMiriam Bruhn Rwanda Green Tea Pricing ReformJana KrajcovicovaBarry Reilly Malawi Streamlining Business Registration Massimiliano SantiniDavid McKenzie Markus Goldstein Francisco Campos