Reflections on the Independent Strategic Review of the Performance-Based Research Fund by Jonathan Adams Presentation to Forum on “ Measuring Research Performance: What are the Options”? 18 September 2008

Presentation Outline 1. Origins, Aims and Implementation 2. PBRF Evaluation Strategy – the review in context 3. Method and approach 4. Key review questions 5. Findings – overall effects; differential effects; unintended consequences; and suggestions for refinement 6. Strengths of the PBRF to protect 7. Dissemination and Utilisation 8. Acknowledgements

Performance-Based Research Fund Origins, Aims and Implementation Origin of PBRF – Key recommendation of the fourth Report of the Tertiary Education Advisory Committee, 2001; detailed design undertaken with the sector and set out in Investing in Excellence. Principles, Metrics, aims Aim – To encourage and reward research excellence within the NZ Tertiary Education Sector Implementation – funding available since 2003 and in $232 million distributed.

PBRF Evaluation Strategy – the review in context Ministerial directive to periodically evaluate PBRF when first introduced 3-phased PBRF evaluation strategy developed in 2003 Phase one – focus on implementation Phase two – focus on emerging effects Phase three – focus on outcomes and cost/benefit Areas of focus in first and second phases of evaluation strategy centre on concerns raised during original policy design Review completes the second phase of evaluation strategy

Method and Approach Initial Design Phase – review of the lessons learnt from evaluation of UK RAE; intervention logic and symposium; consultation with sector on appropriate approach to evaluation Review approach guided by principles independence and credibility of reviewer maximize use of existing secondary data cognisant of participation costs for sector Inclusion of all TEO types for participating in PBRF Cognisant that it is too early to index the systematic impacts of the PBRF Dr Jonathan Adams was the reviewer and his work was supported by a Review Advisory Group Evidence collected via several complementary routes Review report synthesised all material analysed by Dr Adams

What Questions did the Review Address? Effects of PBRF on NZ research base in TEOs Differential effects of PBRF on modes of research, disciplines, institutions and researchers Improvements to consider

Findings – Effects of PBRF on NZ Research Base in TEOs PBRF has been effective in delivering beneficial outcomes Financial – directed more research resources to institutions delivering better research Reputation – increased the quantity and quality of information about relative research quality at institutional and subject level in NZ TEOs Formative – driven improvement in management, culture and awareness and priority given to research These outcomes are expected to result in an overall enhancement in the performance of the NZ research base and in the country’s relative international standing

Findings - Differential Effects of the PBRF on Some Tertiary Sector – ITPs, Wananga and Small PTEs Differential and distinctive missions of TEOs may mean that the PBRF is not the best mechanism for some sectors Dr Adams suggests alternative research support for some tertiary sectors – ITPs, Wananga and PTEs

Findings - Differential Effects of PBRF on Subjects and Modes of Research Does the PBRF model of assessment privilege research subjects, modes and outputs that most easily fit with the traditional western scientific paradigm? PBRF model could work better for humanities / social science disciplines; professional schools and longer-term research with enhanced diversity of panel composition; panels continue to apply appropriate assessment methodology within their subject area; and conceptualising quality whatever the output mode Risky and innovative research – evidence suggests that by rewarding excellence, the PBRF recognises and values such research

Findings - Differential Effects of PBRF on People Available data suggests that the PBRF assessment process does treat different groups of researchers equitably New and emerging researchers have a complex experience. At TEOs participating in both the 2003 and 2006 PBRF census the pool of staff under the age of 35 shrank by 14%. The management of new researchers may affect the long-term sustainability of the tertiary academic workforce and the future welfare of research quality Maori researchers - disproportionately ‘new’ to research; and EPs lower average outcomes but many in fields that attracted lower subject weightings Pacific Peoples’ experiences of the PBRF – data too sparse for sound conclusions

Findings – Unintended Consequences PBRF may create undue focus on staff with established research programmes Some TEOs inappropriately use PBRF quality evaluation results as a staff appraisal substitute Early indications of frustration over lack of significant benefit after raising research quality. The balance between effort and reward could be adjusted to address this.

Findings - Operational Refinements to Consider Who and what is evaluated? Restrict eligibility to a core group of permanent academic staff around whom the research system pivots Consider moving to group as unit of assessment post 2012 Better recognise applied research such as evidence – based policy work How evaluations are conducted – various improvements to panel processes suggested. How different research activities are weighted to ensure emphasis on increasing quality Increase the financial and scoring benefits of the “A” Reduce the weighting for research degree completions Review subject area weightings How results are handled – dissociate scores from staff names and improve reporting.

Findings – Strengths of the PBRF to Protect Additional funding is needed for the pace of improvement required to sustain and increase research excellence PBRF must remain focused on identifying and funding research excellence – avoid adding potentially conflicting goals to its mandate. Developing a mechanism of assessment that focuses on a healthy research environment as well as on excellent individual research

Dissemination and Utilisation of the PBRF Review Report? PBRF Review report is an independently produced source of evidence for use by: PBRF Sector Reference Group Tertiary Education Commission Minister for Tertiary Education Tertiary Education Sector

Acknowledgements Informants PBRF Review Advisory Group Dr Jonathan Adams Contributors of secondary data