1. Solving the Surveyor’s Dilemma: Estimating Future Outcomes from Innovation Programs – the case of the Air Force and Navy SBIR/STTR programs
Robin Gaster (Incumetrics)
Will Swearingen (techlink)
Jeff Peterson (techlink)

2. What is the Surveyor’s Dilemma
Most innovation assessment relies on surveys
Surveys capture outcomes occurring prior to the survey date, but miss subsequent outcomes
As the National Academies noted, many SBIR/STTR projects take a long time to reach the market, and are often in the market for a long time as well
So the surveyor’s dilemma is that surveys inevitably underestimate outcomes
The key question: how much do surveys miss?
Intro – upcoming paper now in review.
Note that NAS tried asking about future outcomes, but discarded results as unreliable

3. Estimating future outcomes
It is possible to estimate future outcomes
BUT this requires a lot of data points; e.g. far more than the Academies studies collected
The best approach is to model outcomes for each cohort of SBIR awardees, and then to apply the model to out years
We use combined data from the TechLink Navy and Air Force Phase II surveys, for awards completed 2000 to 2013
The enormous effort and resources put into these surveys generated more than 7,200 usable responses: enough to build a reasonably robust model.
Bullet 4. note

4. Methodology The study:
Combined data from the Air Force and Navy SBIR studies
Normalized the data using elapsed years - the number of years after the award that the survey was taken
e.g. awards made in 2004 and surveyed in 2014 were combined with awards made in and surveyed in 2016, into a single category: 10 elapsed years (EY) at the time of the survey.
Defined a single outcome metric: commercial sales (including domestic and foreign sales, and sales to the military but excluding R&D sales)
Identified two factors that influence future outcomes:
Post-survey commercialization: projects without sales at the time of the survey that eventually do reach the market
Growing sales: additional sales after the survey date by projects already in the market
Note: Growing sales are more difficult to model because overall success is largely driven by outliers
3. Not likely that there was extensive addition R&D for these projects; licensing data are too spotty for statistical use.
5. these data were log10 transformed before analysis. A linear regression analysis was conducted, with elapsed year as the fixed (X) variable, and the transformed sales data as the random (Y) variable. The analysis produced a significant (p << ) positive slope representing the trend of increasing sales over the time period.

5. Raw data

6. Percentage of total estimated sales not captured by survey (estimated from regression model)
Note: data only allowed us to go back 16 years. However, it’s highly likely that some projects will continue to have sales even beyond that, so these are conservative estimates that might be revised using data from the current TechLink DOD survey

7. Percentage of projects expected to eventually reach the market (>$1,000 sales)

8. Outcomes from modeling
Low numbers of EY3 and EY 5 reflect relatively few projects already reaching the market, and conservative assumptions about the number of projects that will eventually reach the market

9. Increased economic impact (using IMPLAN economic model)
Reported
Estimated Increase
% Increase
Total Output ($B)
92.14
47.34
51%
Jobs
443,138
220,367
50%
Annual Jobs
30,724
15,159
49%
Labor Income ($B)
29.73
14.85
Briefly explain IMPLAN – direct impacts, indirect impacts (sales by vendors who supply SBIR firms), and induced effects (subsequent changes in local economies)
Conclude that this methodology can be applied to other sbir surveys and perhaps to other innovation surveys more generally

10. Conclusions
For ongoing innovation programs like SBIR, much of the impact of innovation programs lies in the future, and impact assessment needs to account for those future outcomes. The Surveyor’s Dilemma is a real problem
Recent studies (e.g. NAS) have noted the problem but failed to address it, hence drastically understating the total impacts of these programs
The methodology presented here offers an appropriate way to estimate future outcomes and to provide as preliminary value for the actual impact of the SBIR program
Integrating into the model data from future studies such as the upcoming TechLink DOD survey will allow further refinement, improved precision, and additional evidence to support estimated impacts.

11. Contacts: Robin Gaster PhD Incumetrics 301-589-5965
Will Swearingen PhD
TechLink
(406)
A pre-publication draft of the paper is available on request