Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium: New Approaches to Assessing the Non-Academic Impact of Social Science London, May 12-13, 2005

My question is, are we having an impact?

Is Research Ready for Action? Medline search ( ] to identify articles stating: a. need more research or need less research Need more161/162 Need less 1/162 b. more questions than answers or more answers than questions More questions 163/166 More answers 3/166 David, AS. BMJ 2002; 323:1462-3

Assessing the Impact of What? The Research Produced? A single research study published in a journal A summary of some research studies written in plain language and posted on a web site A systematic review with targeted dissemination of key messages to potential users A body of research knowledge developed and discussed face-to-face with potential users

Assessing the Impact of What? (cont) The Research Production Process? All the activities of a research commissioning or granting agency (including training) All the activities of a research production facility (e.g. institute, department, university) The activities of a potential research user organization and its staff The entire research regime in a country

Where I Work CHSRFs mission: To support evidence-based decision-making in the healthcare system Our ultimate desired impact is cultural change in the research and healthcare systems Changing the game, not just scoring goals

The Discipline of Objectives & Logic Models Yogi Berra on objectives: If you dont know where youre going, you might not get there Lewis Carroll on logic models: If you dont know where youre going, any road will take you there

Decision Makers Policy Makers Organized Interests e.g. drug companies, professional associations Service Professionals Managers Client & Public Decision Maker Diversity

Why Social Science isnt IBM or General Electric Researchers University- based Stakeholder- based System- based Management Consultants Policy Makers Organized Interests e.g. drug companies, professional associations Service Professionals Managers Patient & Public Decision Makers PROBLEMS SOLUTIONS

Decision Makers Researchers Evidence-Based Decision-Making Research Funders Knowledge Purveyors Receptor Capacity Critical Evaluation PROBLEMS SOLUTIONS Funding and Training Vehicles Priority-setting structures ISSUES & PRIORITIES PRIORITY TOPICS IDEAS RESEARCH EVIDENCE Other Influences Personal Experience, Anecdote, Wants, Interests, Myths, Assumptions, etc. Synthesis and Influence Linkage and Exchange

CHSRFs Objectives 1. To increase health system decision-makers appreciation of the value of research 2. To increase the production of research relevant to the needs of health system decision- makers 3. To increase the availability and acquisition of needed research by health system decision-makers 4. To increase the appraisal and application of needed research by health system decision makers

Increased Appreciation of Value of Research Programs and activities: overall linkage & exchange approach consultations with users for priorities case study presentations of value Measurable outcomes: # of decision-makers participating in research expenditures on research commissioning by system % of employees in system with research training

Increased Production of Relevant Research Programs and activities: priority theme-based program-funding commissioned syntheses on current issues applied training programs encourage university incentives for applied research Measurable outcomes: amount of research in priority theme areas self-reported awareness/use of research syntheses # of graduates with applied research skills

Increased Availability/Acquisition of Research Programs and activities: plain-language research summaries (1:3:25, Mythbusters ) face-to-face exchanges on timely topics creation/support of knowledge networks creation/support for knowledge brokering Measurable outcomes: self-reported awareness of disseminated research self-reported follow-up contact with researchers self-reported use of web-based and other resources for research evidence acquisition (audit computer bookmarks)

Increased Appraisal/Application of Research Programs and activities: training users in research appraisal and application organisational best practices in research use Measurable outcomes: self-reported application of research changes in organisational structures and processes to better accommodate research Increased sense of decision-certainty where synthesised research is available

The Attribution Challenge What, if any, is/was the role of research versus all the other influences on behaviour?

Adapted from Philip Davies, 2005 Program or Intervention Effectiveness Types of Research Evidence Implementation Evidence Organizational Evidence Economic/ Financial Evidence Ethics Evidence Forecast Evidence Attitudinal Evidence Experimental Quasi-Experimental Counterfactual Surveys Admin Data Comparative Qualitative Cost-Benefit Cost-Effectiveness Cost-Utility Econometrics Experimental Quasi-Experimental Qualitative Theories of Change Public Consultation Distributional Data Multivariate Regression Surveys Qualitative

Adapted from Philip Davies, 2005 Types of Research Evidence Program or Intervention Effectiveness Implementation Evidence Organizational Evidence Economic/ Financial Evidence Ethics Evidence Forecast Evidence Attitudinal Evidence Experimental Quasi-Experimental Counterfactual Surveys Admin Data Comparative Qualitative Cost-Benefit Cost-Effectiveness Cost-Utility Econometrics Experimental Quasi-Experimental Qualitative Theories of Change Public Consultation Distributional Data Multivariate Regression Surveys Qualitative Research Evidence

Adapted from Philip Davies, 2005 Combining Research and Colloquial Evidence for Information Research Evidence Professional Experience & Expertise Political Judgement Resources Values Habits & Tradition Lobbyists & Pressure Groups Pragmatics & Contingencies

A Final Zany Idea The impact file - a deductive approach to assessing impact Add photo of impact file

Whats In the Impact File? T F H M & Y D G W - testimonials Altered career trajectory (researcher or decision-maker) Changes in similar organizations programs & processes - lateral impact Better communication of research and its implications - dissemination Awareness of research by decision-makers - acquisition Changed decisions based on research – application Changes in researchers or decision-makers structures and processes - cultural change

Maybe Settling for Second Best is OK! Using an impact file is: Motley, ad-hoc and not comprehensive or systematic Biased to individual not organizational responses Qualitative and potentially non-generalizable But, as Churchill said of democracy: Many forms of government have been tried … No one pretends that democracy is perfect … Indeed, it has been said that democracy is the worst form of government, except all those other forms that have been tried from time to time. House of Commons, 1947

THANK YOU! or