Solving the Surveyor’s Dilemma: Estimating Future Outcomes from Innovation Programs – the case of the Air Force and Navy SBIR/STTR programs Robin Gaster (Incumetrics) Will Swearingen (techlink) Jeff Peterson (techlink)

What is the Surveyor’s Dilemma Most innovation assessment relies on surveys Surveys capture outcomes occurring prior to the survey date, but miss subsequent outcomes As the National Academies noted, many SBIR/STTR projects take a long time to reach the market, and are often in the market for a long time as well So the surveyor’s dilemma is that surveys inevitably underestimate outcomes The key question: how much do surveys miss? Intro – upcoming paper now in review. Note that NAS tried asking about future outcomes, but discarded results as unreliable

Estimating future outcomes It is possible to estimate future outcomes BUT this requires a lot of data points; e.g. far more than the Academies studies collected The best approach is to model outcomes for each cohort of SBIR awardees, and then to apply the model to out years We use combined data from the TechLink Navy and Air Force Phase II surveys, for awards completed 2000 to 2013 The enormous effort and resources put into these surveys generated more than 7,200 usable responses: enough to build a reasonably robust model. Bullet 4. note

Methodology The study: Combined data from the Air Force and Navy SBIR studies Normalized the data using elapsed years - the number of years after the award that the survey was taken e.g. awards made in 2004 and surveyed in 2014 were combined with awards made in 2006 and surveyed in 2016, into a single category: 10 elapsed years (EY) at the time of the survey. Defined a single outcome metric: commercial sales (including domestic and foreign sales, and sales to the military but excluding R&D sales) Identified two factors that influence future outcomes: Post-survey commercialization: projects without sales at the time of the survey that eventually do reach the market Growing sales: additional sales after the survey date by projects already in the market Note: Growing sales are more difficult to model because overall success is largely driven by outliers 3. Not likely that there was extensive addition R&D for these projects; licensing data are too spotty for statistical use. 5. these data were log10 transformed before analysis. A linear regression analysis was conducted, with elapsed year as the fixed (X) variable, and the transformed sales data as the random (Y) variable. The analysis produced a significant (p << 0.0001) positive slope representing the trend of increasing sales over the time period.

Raw data

Percentage of total estimated sales not captured by survey (estimated from regression model) Note: data only allowed us to go back 16 years. However, it’s highly likely that some projects will continue to have sales even beyond that, so these are conservative estimates that might be revised using data from the current TechLink DOD survey

Percentage of projects expected to eventually reach the market (>$1,000 sales)

Outcomes from modeling Low numbers of EY3 and EY 5 reflect relatively few projects already reaching the market, and conservative assumptions about the number of projects that will eventually reach the market

Increased economic impact (using IMPLAN economic model)   Reported Estimated Increase % Increase Total Output ($B) 92.14 47.34 51% Jobs 443,138 220,367 50% Annual Jobs 30,724 15,159 49% Labor Income ($B) 29.73 14.85 Briefly explain IMPLAN – direct impacts, indirect impacts (sales by vendors who supply SBIR firms), and induced effects (subsequent changes in local economies) Conclude that this methodology can be applied to other sbir surveys and perhaps to other innovation surveys more generally

Conclusions For ongoing innovation programs like SBIR, much of the impact of innovation programs lies in the future, and impact assessment needs to account for those future outcomes. The Surveyor’s Dilemma is a real problem Recent studies (e.g. NAS) have noted the problem but failed to address it, hence drastically understating the total impacts of these programs The methodology presented here offers an appropriate way to estimate future outcomes and to provide as preliminary value for the actual impact of the SBIR program Integrating into the model data from future studies such as the upcoming TechLink DOD survey will allow further refinement, improved precision, and additional evidence to support estimated impacts.

Contacts: Robin Gaster PhD Incumetrics 301-589-5965 rgaster@incumetrics.com Will Swearingen PhD TechLink (406) 994-7704 wds@montana.edu A pre-publication draft of the paper is available on request