Measuring the Impacts of Science: Institutional Challenges in a Transforming S&T System Measuring the Impacts of Science Montreal June 2004 © 2004 MPienaar

Overview Approach The National Research Foundation Context : transforming system Unique mission of an intermediary institution Emergence of performance indicators The NRF approach to key performance indicators Challenges © 2004 MPienaar

Approach A tour through the S&T landscape of SA and the route the NRF takes to measure the impact of science © 2004 MPienaar

Mandate of the NRF Support and promote research through funding, human resource development and the provision of the necessary research facilities in order to facilitate the creation of knowledge, innovation and development in all fields of science and technology, including indigenous knowledge and thereby to contribute to the improvement of the quality of life of all the people of the Republic.

Governance of NRF Minister of Science and Technology NRF Board President and CEO Corporate Research & Innovation Support & Advancement (RISA) National Facilities

NRF: RISA © 2004 MPienaar

NRF: Research and Innovation Support and Advancement Information and Strategic Advice Evaluation IT Knowledge Management Research Capacity Development Focus Area Programmes Internl. Science Liaison Building Knowledge Base Support Functions CGA Finance Personnel THRIP (dti) Innovation Fund (DST) Skills Development (DoL) Other (e.g. DEAT) Mission Driven Science for Society Decision Makers Youth (Educators and Learners) SAASTA

National Facilities © 2004 MPienaar

Context: transforming system Measuring the impacts of science and science support of an intermediary body in a transforming system has to account for at least the following factors: Historical chronology and legacy of S&T system Status of National System of Innovation Challenges in NSI and strategies © 2004 MPienaar

Transforming S&T system: Chronology Pre-1990: S&T system – apartheid 1992: IDRC review of S&T system 1994: First democratic government 1996: White paper on S&T 1997: SETI reviews 2002: R&D strategy 2004: Dept of Science and Technology © 2004 MPienaar

Transforming S&T system: legacy First democratic government (1994) inherited a fairly strong S&T Since 1994: activate stagnating economy to –improve the quality of life of all, (esp. the deprived) –broaden the capacity base –become part of the international scene Align new S&T policy with overarching commitments Develop a competent S&T system. © 2004 MPienaar

Status of the National System of Innovation (1) HE institutions: 24 ± –Postgraduate students ± Government labs (SCs): National facilities: 5360 Business sector: ± Researchers / 1000 of labour force: 1.9 GERD as % of GDP: 0.76% Productivity: ISI journals: Patents abroad / million of pop: 2.5 © 2004 MPienaar

Status of the NSI (2): National R&D survey (2001/2002) Business sector performs 53.8% of all R&D undertaken HE sector: 25.3% of all R&D Government laboratories: 19.9% of all R&D Basic research: ± 27% of expenditure African, Coloured or Indian: 36.5% of R&D workforce Women researchers: 35.3% of R&D workforce © 2004 MPienaar Dualistic system Developing - Developed

Transforming S&T system: Current challenge Erosion of S&T base requires clear interventions to reverse this and ensure S&T plays a key role in and for the quality of lives of all our people Wealth creation in modern economies is dependent on innovation, research and development. South Africa must grow its investment in this domain and strategically target resources to meet its objectives Source: Dr Rob Adam, DG: DST

South Africa’s National R&D Strategy The Microeconomic Reform Strategy and dti’s Vision 2014 Accelerating Growth and Development: The contribution of an Integrated Manufacturing Strategy Human Resource Development Strategy Strategic Plan for South African Agriculture National Water Resource Strategy Government’s ICT Initiative Transforming the system: National strategies for economic growth, job creation and social upliftment

Transforming S&T system: NRDS Quality of lifeEconomic GrowthNRDS goals Indicators SET human resources: Correct demographics Science & maths: school level Centres of excellence Innovation: New missions & platforms: Biotech, ICT, Poverty reduction, etc. Effective S&T government system: Alignment with policy objectives IPR © 2004 MPienaar

NSI: Policy and strategy National needs Intermediary institution (NRF) Control: Organisational performance Influence: Stakeholder performance Instrument: Funding Performers of science HEIs and SCs NSI objectives NRF position as intermediary body (1) © 2004 MPienaar

NRF as intermediary body (2) The challenge is to design a performance measurement system that will articulate and translate impacts between different levels Performers Systemic: NSI Institutional policy instruments © 2004 MPienaar

NRF as more than a intermediary body National S&T steering: Policy, business plan & KPI reporting, reviews NRF Needs of and transform- ation in client sector Resources: Slow growth, national priorities Own mission & divergent functions Outcome & impact measurement system Global trends: Strategic science, networks © 2004 MPienaar

Evolution of performance measurement 1996 White paper on S&T 1997 SETI reviews 1998: DST assessed Balanced Scorecard (BC) approach as a framework for KPI reporting in SCs Consultation and participation in development SCs adapted scorecards to fit their specific needs 2000/01: SCs submitted KPI reports Annual submission to Minister © 2004 MPienaar

Generic KPIs for science councils Stakeholder/customer perspective Financial and investment perspective Organisational perspective Innovation and learning perspective Human resources and transformation © 2004 MPienaar

Indicators of Science, Technology and Innovation – NRF Targets (1) IndicatorPresentTargetNRF intervention R&D staff per million of population RSA 480 Korea 2770 Australia by 2012Significant increase in Post-docs and research support staff Patents abroad per million of population RSA 2.5 Korea 779 Australia by 2012Develop NRF policy regarding IPR of publicly-funded research ISI indexed articles per million of population RSA 82 Korea 259 Australia by 2008Alignment with DoE strategy on research output subsidy

Indicators of Science, Technology and Innovation – NRF Targets (2) IndicatorPresentTargetNRF intervention HE enrolment in SET RSA 18% Korea 34% Australia 32% 30% by 2012Create demand pull through significant increase in bursary values PhDs per million of population RSA 18.6 Korea 118 Australia by 2012Treble value of doctoral bursaries Higher Education R&D expenditure as % of GDP RSA 0,07% OECD Avg. 0.38% Australia 0.44% 0.21 % by 2007Creation and investment in Centres of Excellence

NRF: Core missions and priorities Core missions Human resource development Knowledge generation in prioritised areas Utilisation of research results, technology and innovation High quality infrastructure © 2004 MPienaar Strategic priorities Redress & equityAdherence to qualityInternationalisationPositioning of NRFFocus on Africa

NRF: Programmes & Functions RISA Development Programmes Scarce Skills (DoL) STAC Focus Area Programmes CoE THRIP Innovation Fund © 2004 MPienaar Across all programmes & functions: Same core missions & strategic priorities Different interventions & KPIs National Research Facilities SAAO HartRAO HMO SAIAB SAEON iThemba LABS

Use of performance reports by NRF –Creation of integrated organisation –Compliance Public Finance Management Act (1999) –Control, management –Influence i.e. shift in behaviour, relationships, actions of partners –Legitimisation; political acceptance Causality, co-producers, unintended results, etc. © 2004 MPienaar

Challenges for NRF Terminology and definitions Distinguish different levels of complexity Operationalisation i.e. how to “count” concepts Multiple information inputs: reliability & validity Copying more advanced science systems: own developmental requirements? © 2004 MPienaar

Challenges (2) KPIs serves multiple and (potentially) mutually exclusive functions: External: Political legitimisation of science Internal: Institutional management tool

Conclusion The measurement of the impact of science is a scientific endeavour to see things narrowly and deeply. The application in government is to see things broadly and politically.

Thank you…. ….also, in anticipation, for your advice! Acknowledgements Dr RM Adam: Director- General, DST Dr G von Gruenewaldt, Vice-President. NRF Dr HC Marais: Head Secretariat NACI © 2004 MPienaar