1 Improving Measurement and Reporting on the Impacts of Federal Science and Technology Presentation to the FPTT 2008 National Meeting June 3, 2008 André Downs Director General

About the Policy Research Initiative Horizontal policy research that supports strategic policy development and medium-term planning Overseen by a Deputy Ministers Steering Committee Support CCDM policy committees and PCO planning Research portfolio: economic policy social policy climate change environmental scanning (e.g., 2

Summary Diagnostic Conducting policy research with federal departments and agencies on measuring and reporting on the impacts of federal S&T is essential –a key commitment under the federal S&T Strategy announced in May 2007 –federally-performed S&T is considerable (about $5 billion annually) –a significant recent policy literature to draw from 3

The Project in 1 Slide Draw implications from recent policy research, the experience of practitioners, and from original analysis, to suggest how the impact of the S&T performed by the federal government can be better measured and reported, and thereby potentially enhanced over time. A forum or “neutral-space” for federal departments and agencies to identify and explore: concepts that can be applied to draw a more complete picture of the diverse impacts of S&T activities performed by the federal government; options for improving reporting on those activities; and approaches for how to maximize the benefits to Canadians of those activities in relation to costs. 4

Governance Oversight by an inter-departmental steering committee Consensus-driven Preliminary findings tested inside and outside the federal government Public report by February

Project Deliverables 1.Research report on how publicly-funded research organizations transmit technology and knowledge to innovators. (StatCan lead) 2.Identify and assess approaches for additional reporting. (PRI lead) 3.A “think piece” on identifying, describing, and measuring the impacts of federally-performed S&T. (IC lead) 4.Synthesis reports on the market and non-market impacts on Canadians of federal S&T investments. (PRI lead) 5.Four events to present and discuss results. 6.Lessons learned and best practices for assessing and reporting impacts. (NRC lead) 7.Additional work on non-market impacts. (EC lead) 6

1. Public labs and innovators Research report by Frances Anderson from Statistics Canada Applies international definitions and data from the 2005 Canadian Survey of Innovation in Manufacturing to examine how public labs (universities, federal labs, and provincial labs) transmit knowledge and technology to innovative firms 7

1. Public labs and innovators SourceSome importance High importance Clients or customers91.2%*51.9%* Suppliers89.7%*33.4%* Conferences, trade fairs and exhibitions81.7%*14.3%* Universities45.3%3.8% Federal research labs35.2%1.3% Provincial research labs34.0%1.0% Percent of Manufacturing Innovators Indicating that the Source was Important for Innovation Source: Statistics Canada, Survey of Innovation

1. Public labs and innovators Sectors more likely to identify public labs as important: natural resource processing machinery and equipment aerospace The innovators most strongly linked to public labs tended to: be larger; have employees devoted to R&D; be engaged in more innovation activities; collaborate more with public and private organizations; and receive funding from more public and non- conventional private sources. 9

1. Public labs and innovators Percent of Manufacturing Innovators Indicating that their Innovations had an Important Impact, by Impact Area ImpactStrongly linked Not strongly linked Improving job quality43.1%*33.2% Reducing environmental impacts29.4%*18.2% Improving health and safety42.1%*31.5% Meeting regulatory requirements40.7%*26.6% Reducing materials or energy per unit output 42.2%*26.7% Source: Statistics Canada, Survey of Innovation

2. Reporting Definitions Resources are allocated to … … Activities that lead directly to … … Outputs that are intended to achieve … … Outcomes (or Results) that benefit the country. Impacts encompass Outputs and Outcomes 11

2. Reporting Proportion of Federal Technology Commercialization Programs by Type of Indicator in Estimates and Related Documents All programsLarger programs None51%35% Activity*19%27% Output17%19% Outcome13%19 % Total100% * Funded activity described, but no indication provided of its impact Source: PRI research, based on a database prepared by Industry Canada reported in The Impact Group (July 2005), Commercialization Activities of the Federal Government: Program Synopsis. 12

2. Reporting The Relationship among Planning, Reporting, and Impact Type Kinds of Impact OutcomeActivityOutput Planning Planned outcomes ≈ Objectives ≈ Strategic outcomes Planned activities Planned outputs Reporting Results of evaluations, audits and reviews Actual activities Actual outputs 13

2. Reporting Possible Approaches 1.Explicitly incorporate S&T in the Program Activity Architectures (PAAs) of departments and agencies by: A.Treating S&T as an internal service; B.Portraying S&T activities as a single program within each department and agency; C.Describing S&T in terms of activities that in some cases fall under broader programs; or D.Describing S&T in terms of projects (where possible). 2.Expand the annual Statistics Canada survey to collect data in terms of the higher-order PAA categories of departments and agencies. 3.Prepare one government-wide S&T report. 4. Prepare annual S&T reports for each department and agency. 14

3. Think piece on identifying, describing and measuring the impacts of federally-performed S&T Federal S&T Strategy committed to increasing accountability by improving ability to measure impacts of federal S&T investments Considers lessons learned based on past efforts by federal S&T performers to measure impacts Paper designed to help better capture full range of impacts for federally-performed S&T –Analytical framework linking impacts to Strategy’s Entrepreneurial, Knowledge and People Advantages –Currently being finalized 15

4. Synthesis reports The Market Impacts of Federal S&T Investments No recently prepared synthesis has fully reflected the sizeable empirical literature Report will identify: Policy implications based on the synthesis; The methodologies that have been most successful in deriving results; Outstanding policy-relevant research questions; and Suggestions for further research and data development. 16

4. Synthesis reports The Non-Market Impacts of Federal S&T Investments Report will provide: An overview of federal S&T; Conceptual discussion of the non-market impacts of S&T investments; Literature review with a focus on health, safety and security, and the quality of the environment; Empirical evidence on the linkages between market and non-market impacts; Outstanding research issues and knowledge gaps; and Policy implications. 17

5. Project events Preliminary findings in July 2008 Public Science and the Marketplace: Connecting, Measuring and Reporting in September 2008 How do governments support researchers and innovators to develop new ideas and bring them to the marketplace, and how can they measure and report on these efforts? Presentations by Howard Alper, Chair of the Science, Technology and Innovation Council, Ken Carlaw, Richard Lipsey's co-author and collaborator, and Paul Johnston, President and CEO of Precarn, among others Synthesis reports in October 2008 Draft project report in early

Next steps Further work on deliverables Events from July 2008 to early 2009 Final report by February