Presented by Mark Partridge Swank Professor in Rural Urban Policy The Ohio State University October 13, 2013 AUBER Conference Richmond, VA Modeling Economic Development Impacts: We can do better? 1

Introduction/Background The general credibility of impact studies have long been questioned by academics and members of the public. One problem is that they tend to paid for by industry interests. Another possible concern is that they are typically based on purchased software packages that use input- output/SAM methodologies or eclectic packages that add some econometric assumptions about migration and other assumptions about agglomeration economies. See Partridge and Rickman (1998; 2010). One problem is that the methodologies are not best practice in identifying counterfactuals. 2

Introduction/Background 3 AUBER centers try to play more straight with economic impact predictions because of their desire to maintain academic credibility. Yet, Auber centers typically use the same vendor software(s) to estimate impacts. [time constraints] Of course, AUBER affiliates feel pressure to please clients. Can we do better in providing economic impacts that are more credible? I argue yes. Computable General Equilibrium models (CGE), better econometric models, and more attention to external validity are key steps in this process.

Introduction/Background 4 Outline 1) Describe the need to employ external validity. 2) Review typical impact models and appraise their strengths and weaknesses. Review an example from energy of how impact assessments can get out of control. 3) Describe CGE models and appraise their strengths and weaknesses. 4) Describe alternative econometric approaches.

External Validity 5 External Validity: “… is the extent to which the results of a study can be generalized to other situations and to other people. Inferences about cause-effect relationships based on a specific scientific study are said to possess external validity if they may be generalized from the unique and idiosyncratic settings, procedures and participants to other populations and conditions.” Wikipedia, downloaded Sept. 26, I will argue that similar economic shocks that occurred in similar places have more external validity then results from most economic models. The economic impacts of a plant opening in a similar location will be a better predictor of what will happen in your location after a plant opening than the output of a canned impact software package. It is like using Madden 13 to predict the Super Bowl winner when we already know the outcome—the Ravens Won. At the very least, compare (say) IMPLAN results to actual results elsewhere.

Bloomberg News Article 6

Example 7 Let’s use my prior discussion of external validity to assess whether such claims make any sense? What is a reasonable case to apply external validity for a large oil boom?

Picking Winners—”This Time is Different” 8 What about fossil fuels? Both Presidential candidates trumpeted recent innovations as cornerstones to their job creation strategy. Ancillary is the Keystone Pipeline and job creation. Massive media campaigns in the energy industry trumpeting job creation.

Bakken Oil Production Source: North Dakota Industrial Commission, Department of Mineral Resources, Oil and Gas Division Montana Department of Natural Resources and Conservation, Oil and Gas Division Over 700,000bbl by Dec 2012, probably nearing 800,000 today

Bakken Oil Counties defined by direct employment

Bakken Media Reports—seekingalpha.com

Unemployment Rate Source: U.S. BLS

Bakken Employment Since 2002 Bakken employment has increased from 118,500 to 167,800 in 2012 (an increase of over 49,000) Source: U.S. BLS, QCEW

External Validity and California 14 If North Dakota’s Bakken region only created 49,000 jobs with a massive oil boom, it is hard to fathom how an oil boom in California, with dense population and environmental regulations that would slow development, could create 2.8 million jobs over the period. Use of external validity can avoid embarrassing forecasts.

Traditional Impact Studies: Certainly not best practice. 15 Discussion follows Partridge and Rickman (1998;2010). Traditional impact studies are typically of two types: Most popular are input-output/SAM oriented—e.g. IMPLAN. Eclectic models that incorporate input-output and econometric modeling—e.g., REMI. Why? 1. Well known. 2. Relatively easy to use and IMPLAN is especially low-cost. 3. Especially with IMPLAN, the modeling can be done very fast.

Traditional Impact Modeling. 16 The problems with traditional impact analysis are well known. (I will emphasize the I-O/SAM concerns). The founding father of IO, Wassily Leontief, was interested in Central Planning and the most widely used IO models lack prices. Impact studies of direct and indirect effects are typically over-estimates of new job creation (jobs supported) and academic regional economists have not viewed them as anywhere near best practice for decades. Edmiston (2004) is a good ex post evaluation of economics impacts. Very small effects are consistent with the natural/man-made disaster literature (Davis and Weinstein, 2002; Xiao, 2011). Long-run growth path.

Traditional Impact Modeling 17 A typical impact study should tell how many jobs are ‘supported’ by an industry, not how many jobs it ‘created.’ At the very least, this needs to be much better explained. Also the timing of jobs needs to be more transparent— e.g., the temporary nature of construction vs. other jobs. Confidence intervals should be reported. The general problem is that the IO models are not linked to spatial equilibrium modeling and do not reflect the displacement effects from a positive shock and they do not account for changes in entrepreneurial behavior. There are approaches that account for these concerns in the natural experiment literature and matching. More on this below.

Traditional Impact Modeling 18 One suggestion to provide more honesty to traditional impact modeling is that confidence intervals be reported. Vendors should incorporate simple Monte Carlo based bootstrap procedures to estimate the standard errors of the estimates. This is something that has been done with CGE models (Partridge and Rickman, 1998, 2010). It would be doable for vendors to provide. Another suggestion is to use actual data from similar shocks to assess whether the software is producing reasonable results. External validity.

Traditional Impact Modeling 19 Example of using actual data and external validity for traditional impact results. Ohio “Shale Energy Boom.” Kleinhenz & Associates (2011) funded by the industry predicted over 200,000 jobs in Ohio by Used REMI. Ohio Shale Coalition (2012) predicted 66,000 jobs by Used IMPLAN Weinstein and Partridge (WP) (2011) predicted 20,000 by WP noted that Pennsylvania had a boom that proceeded Ohio’s and given the similarities of the two states, used the direct employment effects in PA for the first 4-5 years of their boom—which they generously estimated at 10,000 (counted some indirect effects or double counted). Then using a multiplier of 2, WP estimated 20,000 jobs for Ohio. Reporting actual data, through Qtr , employment growth in the strong shale regions is imperceptibly different from the rest of the state (College of Urban Affairs, Cleveland State University, 2013). This suggests WP appear to be correct.

CGE Modeling 20 Partridge and Rickman (2010) are optimistic that regional CGE models can be constructed with realistic migration, capital location, and commuting effects. P&R offer suggestions to further improve their accuracy in which they too often mimic international models. CGE models flexible enough to analyze many impacts such as land use, tax policy, income distribution, and employment shocks. My first 2 publications constructed an early regional CGE model to assess tax policy (Morgan et al., 1989; Mutti et al., 1989). CGE models have the key advantage of using a structural framework make them. As someone who has built these models and a reviewer-editor for dozens of CGE papers, their results are quite robust to even non marginal changes in key parameters. Why, offsetting price changes allow the model to remain stable.

CGE Models 21 Case closed, let’s switch to CGE models. Right? There are some constraints. The biggest is that there is still no canned programs and they are not easy to build. Yet there are wonderful examples of regional CGE models that could be employed: 1. AMOS at the University of Strathclyde. Harrigan et al., Colorado State University Model—e.g., Burnett et al., 2012; Cutler and Davies, 2010; Schwarm and Cutler, Federal-F for Australia-- Giesecke and Madden, 2003a, b. 4. The late David Holland at Washington State had a model of the Washington state economy—Cassey et al., The take away is that I urge you to look into CGE modeling for your work.

Econometric Solutions 22

Econometric solutions 23 INDMIX could be constructed for individual sectors to get multipliers by sector. Marchand (2012) describes sectoral multipliers. The outcome is that you could create multipliers based on real world behavior (with confidence intervals) rather than being the outcome of model that doesn’t produce standard errors.

Econometric Solutions 24 Simple matching of places receiving the treatment and those who did not can be employed. It could employ regression approaches to control for observables or even simple t-statistics on differences. In this case, the counterfactual is similar places that had similar economic characteristics and trends prior to the treatment. This can be done with statistical software and identifying treatment as (for example) places that had a big change in employment in a certain sector. A really simple example is what we did to assess the early impacts of the Pennsylvania shale boom. We found modest effects on employment and large impacts on income.

25

PA Counties considered in our simple difference in difference counterfactual

Source: U.S. Bureau of Economic Analysis, REIS Data, Downloaded Oct. 7,

Source: U.S. Bureau of Economic Analysis, REIS Data, Downloaded Oct. 7,

Source: U.S. Bureau of Economic Analysis, REIS Data, Downloaded Oct. 7,

Source: U.S. Bureau of Economic Analysis, REIS Data, Downloaded Oct. 7,

Econometric Approaches 31 There are more complicated matching approaches but finding simple treatment cases and comparing to similar non- treatment cases provides real-world predictions of what to expect.

Conclusions/Summary 32 Economic Impact modeling often uses non-structural models to estimate impacts. These models are often criticized for producing overestimates of the economic impacts. I described some ways to improve the estimates and enhance the credibility of their predictions. 1. Use external validity in confirming forecasts. 2. Provide confidence intervals on the estimates of models using Monte Carlo bootstrapping. 3. Provide a better interpretation of the timing of multipliers and the meaning of jobs supported versus net job creation.

Conclusions/summary Use structural CGE models which have a clear counterfactual. 5. Use econometrics approaches such as estimation of multipliers for total employment growth or employment growth by sector and matching approaches. Matching in particular is linked to a clear counterfactual. However, it is clear that we can improve from the status quo with more attention on actual data from actual economic impacts can produce more credible estimates.