1. Measuring Research Engagement and Impact in the Mathematical Sciences
Kerrie Mengersen

2. “Mathematical sciences is a broad church”
1. Who are “we”?
“Mathematical sciences is a broad church”
Deloitte 2012 “Measuring economic benefits of mathematical sciences research in the UK”
Mathematical science research
Mathematical science occupations

3. “Mathematical sciences is a pipeline”
1. Who are “they”?
“Mathematical sciences is a pipeline”

4. 2. What is “impact”?
1. the action of one object coming forcibly into contact with another
synonyms: collision, crash, smash, clash, bump, bang, knock, jolt, thump, whack, thwack, slam, smack; contact
"car parts were spread by the impact over a wide region"
2. a marked effect or influence
“Turning outputs to outcomes”
Siu-Ming Tam, Australian Bureau of Statistics
“It changed our bottom line. We couldn’t have done it ourselves”
Gerard Davis, Australian Agricultural Company

5. Deloitte 2012 Report
Broad impact: societal, can sometimes be quantified
Direct impact: immediate changes
Indirect impact: associated changes
Induced impact: spending by households that result in changes to jobs and GVA

6. “It takes [at least] two to engage”
2. What is engagement?
“It takes [at least] two to engage”
ATSE Report (March 2016)
“Engagement describes the interaction between researchers and research organisations and their larger communities/industries for the mutually beneficial exchange of knowledge, understanding and resources in a context of partnership and reciprocity”

7. 2. What is engagement and impact?
“Engagementandimpact is not a single word”

8. 3. How do we measure it? Deloitte 2012: Economics
Number of jobs and Gross value added (GVA)
measure of value of goods and services produced by business, industry sector or region of the economy
Access Economics Report on Health Research (2008)
Return on investment (ROI)
gain in wellbeing (non-financial measure, DALY), relative to what would otherwise have been
Wellbeing metric: DALY (non-financial)
Value of DALYs averted converted to a dollar equivalent using willingness to pay estimates

9. 3. How do we measure it? Traditional (Mode 1) view “Single lens” view
ATSE Report (2016) REA Metrics
Research per FTE; share of national engagement (income); engagement intensiveness (university’s total revenue from continuing operations)
A broader view
Traditional measures
+ Mode 2 measures
+ Peer review

10. “A broad evaluation of research”
3. How do we measure it?
“A broad evaluation of research”
2011 Group of Eight review
2014 UK Research Excellence Framework
US National Science Foundation

11. “From anecdotes to metrics”
3. How do we measure it?
“From anecdotes to metrics”
2014 UK Research Excellence Framework
US National Science Foundation
2011 Group of Eight review
input measures; output measures and benchmarking; expert review; anecdotes; case studies; cost-benefit analyses; hindsight studies; surveys; economic models; econometric analysis
RAND Europe Impact Finder (UK)
tracks the impact of products with digital object identifiers in blog posts, tweets and media
ImpactStory
tracks articles, data sets, software, blog posts, posters and lab websites by monitoring citations, blogs, tweets, download statistics and attributions in research articles.
Journals
“tweetations” as an indicator of highly cited articles.
Research Gate
overall score based on unpublished research

12. 4. Where to from here? Thorough review
Develop indicators, measures and metrics based on and targeted to our own discipline
Identify high value measurements (via Value of Information Analysis)
Close links between indicators, measures and metrics, and objectives and decisions (why collect this information and how will it be used?)

13. If any discipline can do this, we can!
Key messages
Mathematical sciences is a broad church
Engagement and research is a pipeline
Engagementandresearch is not one word
A broad measurement system is required
If any discipline can do this, we can!