1. Business and Systems Aligned. Business Empowered. TM Three Approaches to Measuring the Economic Impacts of RTD Toronto, ON. October 29, 2005 Presentation to the CES/AES Conference A. Dennis Rank and Frederick Kijeck

2. 1 © 2005 BearingPoint LP Overview Three studies of three programs. Programs had very different goals, structure, operation, and impacts. Three different challenges, and solutions. Thus three entirely different methods. All studies measure benefits from “national perspective”

3. 2 © 2005 BearingPoint LP Three Programs

4. 3 © 2005 BearingPoint LP Three Studies

5. 4 © 2005 BearingPoint LP Precarn Challenge and Solutions

6. 5 © 2005 BearingPoint LP CSRS Challenges And Solutions

7. 6 © 2005 BearingPoint LP HPPM Challenges And Solutions

8. 7 © 2005 BearingPoint LP Precarn Methodology Bottom-up Partial B-C: Net benefits = (Gross benefits of “big winners”) – (Implementation costs). Total program costs = (Grants and contributions) + (Overhead costs) + (Costs to all other parties). All benefits and costs discounted. Net Present Value (NPV) NPV = (Net benefits of “big winners”) – (Total program costs to all parties). Benefit: Cost ratio (B/C) B/C = (Net benefits of “big winners”) ÷ (Total program costs to all parties).

9. 8 © 2005 BearingPoint LP Precarn Findings Commercial net benefits, cost savings from 6 Big Winner Projects, @ 5.5% discount rate: Lower bound: — Net Benefits = $312 million — NPV = $83.7 million — B/C ratio = 1.37 Upper bound: — Net Benefits = $1.3 billion — NPV = $1.1 billion — B/C ratio = 5.77

10. 9 © 2005 BearingPoint LP Precarn Lessons Learned Crucial to know projects and impacts, likely “big winners”. One technology, one firm, one market always easiest to deal with – makes partial B-C an obvious choice. But ONLY if can get confidential $ data. Clients still need “education” about B-C.

11. 10 © 2005 BearingPoint LP CSRS Methodology Top-down Gross Value Added (GVA): — CSRS influence estimated (Low to High) for 8 sectors & 30+ sub-sectors — Influence quantified by proxy % index. — $ Influence per sector = (Sector GVA) x (Proxy % index). — Total influence of CSRS on Canada = Sum of GVA influences per sector. Bolstered by other study activities: — User surveys & interviews, focus groups, policy analysis, review of international and Canadian literature, qualitative case studies, etc. — Benchmarking to NSRS elements in other countries. — NSRS comparison indices in benchmark countries on $/capita, $/sq. km., $/GDP.

12. 11 © 2005 BearingPoint LP CSRS Findings CSRS “influences” $60B - $90B of Canadian GVA, or about 6% – 9% of total. Backbone of many industrial, government, and research applications, though many users know little about NSRS or CSRS. This figure (as far as possible) modeled on geodetic impacts alone.

13. 12 © 2005 BearingPoint LP CSRS Lessons Learned Top-down sometimes only way to go. But top-down methods difficult to understand, open to interpretation – need other lines of evidence (the more quantitative the better). ID of incremental value-added is harder than in bottom-up methods. Entrenched suspicions difficult to overcome.

14. 13 © 2005 BearingPoint LP HPPM Methodology Qualitative case studies of interesting commercial and policy impacts. Follow-up case studies focused on very specific, concrete long-term impacts first identified by ex-HPPM expert. Impacts confirmed through modified peer review re: research quality, relevance, and impacts.

15. 14 © 2005 BearingPoint LP HPPM Findings HPPM believed to be a commercial failure: No direct commercial applications of technologies (e.g., application of patents). Partially due to nature of technical targets – HPPM would not support applied R&D, and difficult to commercialize R&D not in partners’ product line. Scientific importance actually quite high: HPPM showed better to improve existing membranes than develop new ones. Understood physics and chemistry re. how and why membranes worked. These were breakthrough concepts for all membranes – not just polymers, & not just separation membranes.

16. 15 © 2005 BearingPoint LP HPPM Findings (cont’d) High long-term commercial impacts: New products (e.g., nanofilters) related to HPPM topics, or non-polymer materials (e.g., ceramics), or other types of polymers, or in entirely new fields. Linked – often quite indirectly – to HPPM ideas. Some firms world leaders & “first to market” partially because of HPPM. Impossible to obtain sales figures for these products and processes. Impossible to judge exact role of HPPM, but good reasons to believe it was at least moderately important in these later developments. Also impacts on competencies, researchers’ careers, & innovation networks. Possible impacts on patent laws, R&D policies, & design of R&D programs.

17. 16 © 2005 BearingPoint LP HPPM Lessons Learned Tracking long-term impacts “easier” when considerable time elapsed. Need to be very specific and explicit when trying to ID and track these impacts – they are not well understood by participants. Need external, independent confirmation. Client and analyst need to be flexible re: methods.

18. 17 © 2005 BearingPoint LP Overall Lessons Learned 1. No “magic bullet” – cookie cutter approach won’t work. 2. Difficult to overcome entrenched attitudes and suspicions – many lines of evidence, lots of documentation help, but don’t entirely solve problem. 3. Very useful for program to track “big winners” & users, even for top-down studies. 4. Multi-step process for long-term knock-on & spin-off benefits works, but need to ID very explicit impacts – and don’t ignore non-technology transfer benefits. 5. Follow your nose, be flexible – don’t ignore oddballs.