NIFU STEP studies in Innovation, Research and Education Peer review of impact? Options and challenges Liv Langfeldt RCN 14 April 2008

NIFU STEP studies in Innovation, Research and Education 2 Impact of research? Science impacts Technology impacts Economy impacts Culture impacts Society impacts Policy impacts Organisation impacts Health impacts Environment impacts Symbolic impacts Training impacts (Godin and Doré 2006/OECD 2008)

NIFU STEP studies in Innovation, Research and Education 3 Challenges in assessing impact of research Extended effects/combined effects (no linear model)  Complex transfer mechanisms/interplay  National, sectoral and disciplinary spillovers Time lag:  Often both ex ante and ex post peer review assesses potential, not attained, impact  What can replace the linear model as bases for measuring potential impact? Who are the peers or experts for assessing extra- scientific impact?

NIFU STEP studies in Innovation, Research and Education 4 Impact focus may include a wide variety of initiatives Examples from the Norwegian context  Performance based budgeting in higher education and hospital, to be introduced also for the research institutes (“Tellekanter”)  Programme development/formulation (ex ante)  Project selection (ex ante)  Programme evaluations  Costumer surveys on user directed research  Evaluation of research fields  RCN projects’ final reports  Communication of findings/results from projects (RCN efforts in facilitating impact)  Foresight studies (ex ante)

NIFU STEP studies in Innovation, Research and Education 5 Performance based budgeting in Norwegian higher education Education indicators: about 25 percent of core funding Research indicators: about 15 percent of core funding  Doctoral candidates: weight 0.3  EU research funding: weight 0.2  RCN research funding: weight 0.2  Scholarly publications: weight 0.3  Includes all sources in all fields. Publication sources divided into two levels giving different scores.  Annual meetings in each discipline to determine and revise “level 2” under the guidance of the Norwegian Association for Higher Education Institutions.

NIFU STEP studies in Innovation, Research and Education 6 Project level impact: Ex ante Scientific impact: the typical peer review task  Project selection: Guessing potential impact based on past performance and project description  Different peers may have different assessments/opinions about impact:  The likeliness/conditions for impact  The importance of the potential impact (scientific value/relevance)  The relative societal importance of the research topic (and the legitimacy of such concerns) If we do not believe in the linear model and are unable to separate impacts of specific projects, on project level ex ante estimates of extra-scientific impacts are complex/less meaningful.

NIFU STEP studies in Innovation, Research and Education 7 Project level impact Links between ex post and ex ante assessments:  Ex post assessments usually related to general scientific merits of the group/PI, not the specific projects.  Ex post assessments of projects may have little implications for the evaluees – only one of many evaluations of their activity  Publications: basis for assessing results and for impact on career (awards, positions)  Extra-scientific impacts have little effect on academic careers? Options  Link project selection more directly to former projects’ final reports, programme and/or field evaluations?  Study how past extra-scientific impacts influence project selection? (and vice versa)

NIFU STEP studies in Innovation, Research and Education 8 Programme level impact Ex ante impact assessments  Negotiations including a wide set of stakeholders defining the programme Ex post impact assessments: challenges  Evaluation at the end of a programme is still mainly ex ante with regard to impact  Costumer surveys  Impacts for a limited group  Mainly ex ante/potential impact

NIFU STEP studies in Innovation, Research and Education 9 National level impact Ex ante assessments of relevance/impact  Policy formulation  Budget allocations  Various stakeholders involved Ex post assessments  Should evaluations of research fields include impact assessments?

NIFU STEP studies in Innovation, Research and Education 10 Example: Evaluation of Norwegian development research ToR: Assess whether the research was utilized  One of about 13 questions for the international expert panel Sources on extra-scientific use/impact  Self assessment: “Examples of results/impacts of your development research”  Interviews with users Conclusion  The research has policy relevance and a high proportion of the research is directed at user needs  Mapped different ways of use

NIFU STEP studies in Innovation, Research and Education 11 Possible reactions to a demand for more impact measurements Reframe in impact terms what is already done Rethink timeframe for evaluations/the notion of an evaluation Rethink the role of peer panels and other stakeholders in evaluations Separate panels/assessments  The relevant societal/non-academic impacts of the research  The scientific success and excellence of the research  The connections still the most interesting?

NIFU STEP studies in Innovation, Research and Education 12 Models for performance based funding of research institutions Panel evaluation, ranking Bibliometric models All disciplines; all publications All disciplines; selected publications Selected disciplines; all publications Selected disciplines; ISI data N GB B AUS S SF DK