Examination of Multi-department (Horizontal) S&T Programs Presented to:AEA Evaluation Conference November 2, 2006 Presented by:Steve Montague and George Teather Performance Management Network Inc. Tel:(613) Fax: (613)

Multi-departmental “Horizontal” Programs Horizontality or integrated program delivery is gaining ground in program design as a means of reducing Departmental turf wars and “silos” Provide a means of addressing higher level government objectives beyond the scope and capability of a single Department To date, horizontal has had several interpretations –Common program theme, each Department receives own funding to address common theme within Departmental mandate, but no requirement to work together ( e.g. genomics R&D) –Common program theme, most projects involve researchers from several government Departments, each bringing own expertise to address a common objective (e.g. Panel on Energy R&D) “Program” is at a higher level, and requires additional strategies for evaluation 2

3 Federal Energy and Energy Related Environmental S&T Investments (FEESTI) Natural Resources Canada is responsible for Canada’s energy policies and programs related to oil and gas production, renewable and other electrical energy production and energy in general NRCan has a number of energy related programs and works with a number of other departments, agencies and foundations on energy related initiatives Energy and environmental issues are closely intertwined (creating energy from nonrenewable sources creates CO 2, other gases that contribute to greenhouse gases (GHGs). International implications (Kyoto, etc.) NRCan recently commissioned a high level review of the range of energy and energy related environmental S&T investments, primarily within the federal government NRCan funded programs span much of middle part of the innovation spectrum programs Focus on those funded by NRCan (primary area of influence)

4 Scope Energy and energy-related environmental S&T Organizations involved – Category 1 NRCan programs, Category 2 Departments receiving NRCan funding, Category 3 other departments and agencies TEAM T&I PER D NRC IRA P AECL PWG SC IC AA FC EC DFO CCAP INAC HC TC OCS CFS CMH C MMS ESS NSER C Category 1 Category 2 Category 3 SDTC CET C Ener gyIN et Reach of FEESTI

5 Examination In what manner, and to what extent, do the Federal energy and environment S&T programs complement, duplicate, overlap or work at cross-purposes with each other? How effective have program mechanisms been in promoting innovation diffusion / transfer? To what extent are the financial resources attached to the Federal energy and environment S&T programs allocated and managed effectively? Three aspects Complementarity, overlap and duplication Programs as innovation support mechanisms Management of resources

6 Source: Rostum, H., The Information Imperative: A framework for measuring impacts of STM information services and STM information organizations, 2003 Elements of the Knowledge-based Economy and the Innovation System (Simplified)

7 (Source: PERD and T&I Comparison 2004) A Traditional View of the Innovation Spectrum

8 Operational Your operational environment You have direct control over the behaviours within this sphere Behavioural Change Your environment of direct influence e.g. people and groups in direct contact with your programs, staff (i.e. clients, target audience, co-delivery partners State Your environment of indirect influence e.g., Industrial sectors, government decision makers, other communities of interest where you do not make direct contact Spheres of Influence* Time *reference S. Montague, 8

9 NRCan Direct Influence NRCan Indirect Influence Broader Communities / Canadians Increased awareness, understanding, acceptance and public confidence in energy and environmental technology and compatible technologies Trades persons, companies, universities, community colleges and other private and public institutions build skills and energy technology capacity Relative cost-effectiveness of different energy alternatives is improved Adoption of technology beyond funded groups (first purchase, spin-off technologies, innovation, diffusion) An integrated supply of compatible technologies is developed Strengthened industrial capability in Canada – critical mass created in key communities Sustained societal energy, security, economic and environmental gains Immediate Outcomes (Direct) Related Public S&T Funders Information exchange Consultation, coordination, collaboration Project / initiative development refinement and management Funding (resource) management Intermediate Outcomes (Indirect) Final Outcomes Inputs (Resources) OutputsActivities FEESTI Program Funders Information exchange Consultation, coordination, collaboration Project / initiative development refinement and management Funding (resource) management Private Sector S&T Funders Information exchange Consultation, coordination, collaboration Project / initiative development refinement and management Funding (resource) management NRCan Control The results chain is actually an interconnected system of relationships leading to technical, economic, social and environmental impacts (see yellow shading). The Reach and Results of (NRCan Funded) Federal Energy and Environment S&T Investment Energy Sector Communities (S&T Performers and Users) Information exchange Awareness, understanding, acceptance Cooperation, coordination, collaboration of public, private and NGO groups Projects successfully implemented Successful technical and commercial results Development of new organization / sector capabilities Codes, standards and guidelines identified and developed Policies and legislation changed on the basis of related S&T work

10 User Communities Standards, Regulatory Organizations Research Organizations Partner Groups Individuals in the social setting Flow of new knowledge Formal organizational structures Informal structures Key A Social Interaction View of Innovation Source: Adopted by Przybylinski et al., Temporary Organizations for Collaborative R&D: Analyzing Deployment Prospects, 2000, from Havelock and Havelock. Derived from U.S. DOE and U.S. National Institute for Standards and Technology (NIST), A Toolkit for Evaluating Public R&D Investment Models, Methods and Findings from ATP’s First Decade Ruegg and Feller, July 2003, p 132

Panel on Energy R&D Funded Network Based on Reported Total Funding by Partner Node 11

12 Levels of Interaction LevelsPurposeStructureProcess Networking (1)  Dialog and common understanding  Clearinghouse for information  Create base of support  Non–hierarchical  Loose/flexible link  Roles loosely defined  Community action is primary link among members  Low key leadership  Minimal decision making  Little conflict  Informal communication Cooperation or Alliance (2) Match needs and provide coordination Limit duplication of services Ensure tasks are done Central body of people as communication hub Semi–formal links Roles some what defined Links are advisory Group leverages / raises money Facilitative leaders Complex decision making Some conflict Formal communications within the central group Coordination or Partnership (3)  Share resources to address common issues  Merge resource base to create something new  Central body of people consists of decision makers  Roles defined  Links formalized  Group develops new resources and joint budget  Autonomous leadership but focus in on issue  Group decision making in central and subgroups  Communication is frequent and clear Coalition (4)  Share ideas and be willing to pull resources from existing systems  Develop commitment for a minimum of three years  All members involved in decision making  Roles and time defined  Links formal with written agreement  Group develops new resources and joint budget  Shared leadership  Decision making formal with all members  Communication is common and prioritized Collaboration (5)  Accomplish shared vision and impact benchmarks  Build interdependent system to address issues and opportunities  Consensus used in shared decision making  Roles, time and evaluation formalized  Links are formal and written in work assignments  Leadership high, trust level high, productivity high  Ideas and decisions equally shared  Highly developed communication Source: Hogue, Teresa Community Based Collaborations

13 Observations Need for comprehensive, coherent strategy accepted by all participants Ensure complementarity and collaboration, not duplication Funding cycle consistent with type of program (need for funding certainty) Comprehensive and consistent approach to program management and performance measurement among organizations Need to develop strategy to manage the network (not command and control)

Generic Program Logic Model Program Objective: high level strategic purpose HOW? WHO / WHERE? WHAT do we want? WHY? ResourcesReachResults Program / Service Delivery Client Management Policy & Issue Management Financial Management Human Resources Management Asset Management Program deliverables Policy guidelines, regulations Communications -plans - internal communications - promotion - info transfer - consultations - meetings/events Funding Service Outputs Primary Targets (clients, ultimate beneficiaries) Co-delivery Agents Other Stakeholders activitiesoutputs users / clients / co-deliverers / beneficiaries direct outcomes intermediate outcomes ultimate impacts Client Service - addresses needs - meets / exceeds expectations - service quality Behavioral Influence -awareness -understanding -attitude / perception -support New knowledge Improved capability Improved decision making Target group changes in behaviour / other outcomes Sector / Industry / Regional Impact Economic/ Environmental/ Societal Impact Contribution to organizational objective 14