1. 1 www.pmn.net The Use of Surveys in Determining Outcomes of Research, Technology and Development Programs Presented at:Joint CES / AEA Conference Presented by:Suzanne Lafortune, Partner Performance Management Network Inc. Date:October 26, 2005 Tel:(613) 236-2320 Fax:(613) 236-8644 Email:suzanne.lafortune@pmn.net

2. Results & Resources versus Reach Source: Steve Montague, The Three Rs of Performance: Core concepts for planning, measurement, and management, Performance Management Network Inc., 1997, p 120. The ‘clustering’ of services or activities by results, resources per user, and reach can provide new perspective. Intensive Problem Solving / R&D Technical Specialist Support Education Information / Advice Awareness Building Reach Results + Resources per user education and communication (workshops, speaking, trade shows) routine advisory dissemination of voluntary codes mass media Information products policy development program funding enforcement and prosecution specialized advisory investigations inspections and monitoring www.pmn.net

3. Frequency and Type of Measurement Reach Results & Resources versus Reach Results + Resources per user Intensive Problem Solving / R&D Technical Specialist Support Education Information / Advice Awareness Building Source: Steve Montague, The Three Rs of Performance: Core concepts for planning measurement and management, Performance Management Network Inc., 1997, p.20 Methods OngoingForm. EvalSum. Eval Documents Databases Interviews Surveys** Case studies All Documents Databases Surveys* Interviews Surveys** Case studies All Documents Databases Surveys* Interviews Surveys** Documents Databases Surveys* Interviews Surveys** Documents Databases Surveys* Interviews 3 * Project related surveys ** Periodic surveys related to more in-depth impacts

4. 4 Why Use Surveys for Formative / Early Evaluations of R&D Programs? u Relevance of program design to R&D needs u Satisfaction with ways in which program is designed and delivered ä Eligibility criteria ä Application process ä Use of advisory boards ä Deadlines ä Time of year (academic community versus private sector) ä Speed of decision making ä Feedback / advice / staff communications and support before project / in early phases of project ä Technical competence / expertise of staff

5. 5 Why Use Surveys for Formative / Early Evaluations of R&D Programs? u Early outcomes – before project ä Awareness building ä Strategic alliances with partners ä Leveraging u Early outcomes – during project ä Active participation of partners ä Knowledge gained ä Job creation / maintenance within organization during project u Early outcomes – at the very end of the project ä Improved technical capability ä New / improved product / process / technology / service u Project incrementality ä Important to measure early on, particularly for longer term R&D projects

6. 6 Why Use Surveys for Summative Evaluations of R&D Programs? u Relevance of program implementation to R&D needs u Satisfaction with ways in which program was implemented ä Feedback / advice / staff communications and support ä Technical competence / expertise of staff ä Reporting requirements ä Payment process

7. 7 Why Use Surveys for Summative Evaluations of R&D Programs? u Intermediate outcomes ä Impact on recipient organizations Ô Revenues / sales Ô Productivity Ô Sustained / long term jobs Ô Competitiveness / increased market share Ô Increased profitability Ô Cost savings Ô Exports / new client base Ô Diversification Ô Expansion Ô Etc. ä Impact on industry Ô Spin offs Ô Technology transfer

8. 8 Why Use Surveys for Summative Evaluations of R&D Programs? u Long term outcomes ä Impact on the economy / on society Ô Socio-economic benefits – Reduced consumer costs – Employment – Improved quality of life – Etc. Ô Environmental benefits Ô Improved energy efficiency Ô Health and safety benefits Ô Etc.

9. 9 Types of Target Groups to Survey for R&D Evaluations u Non-recipients ä Formative evaluation – relevance to needs, satisfaction, incrementality ä Summative evaluation – relevance to needs, incrementality u Recipients u Project partners ä Formative evaluation – impact of program on participation, early project outcomes / results on partner organizations ä Summative evaluation – extent of participation, long-term collaborations, other outcomes / results on partner organizations u Beneficiaries ä Summative evaluation – impacts of projects on beneficiaries