1. Capturing the impact of research Briony Rayfield

2. organisation individual grant funding scheme / programme overall strategy Seeking ‘impact’: levels of monitoring & evaluation process, learning & strategy outputs focus

3. Strategic Plan & Assessment Framework OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

4. Monitoring progress: WT’s key indicators OutcomesKey indicators of progress Discoveries Applications Engagement Research leaders Research environment Influence 1. significant advances in the generation of new knowledge 2. contribute to discoveries with tangible impacts on health 3. contribute to the development of enabling technologies, products and devices 4. uptake of research into policy and practice 5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach 7. develop a cadre of research leaders 8. evidence of significant career progression among those we support 9. key contributions to the creation, development and maintenance of major research resources 10. contributions to the growth of centres of excellence 11. significant impact on science funding & policy developments 12. significant impact on global research priorities and processes

5. Key data sources grant progress reports (online tool, end of grant/review) bibliometric analyses (normalised citation impact, F1000, ORCID) IP data case studies surveys career tracking other outputs (e.g. guidelines, alt metrics (‘use’ of research) press releases economic analyses

6. M & E :approach be prospective: build m & e frameworks in from the start ensure access to key data, information & resources consider option for trends & benchmarks contribution not attribution keep it real: proportionate & practical – measures can evolve flexible – learning is iterative & can be part of the process

7. Malaria portfolio review to identify the key landmarks and influences on the research landscape over the last 20 years at a subject, portfolio level speculate on the future direction of the field and consider where there maybe opportunities for Trust strategy and funding Malaria, Human Functional Brain Imaging, Human Genetics, Population and Public Health key ingredients:  landscape analysis (funding, bibliometrics, landmarks)  expert ‘peer review’  retrospective and prospective

8. ⁺Stock take of where we are ⁺Checking our funding fits with our strategy ⁺Use of external experts to evaluate impact ⁺Timelines (human genetics and brain imaging) been used by others for education resources ₋Funding landscaping very challenging ₋Actions on recommendations ₋Resource intensive Malaria portfolio review

9. Key aims 1. Strengthen institutions 2. Training of individuals 3. Equitable networks 7 consortia awarded involving 70 institutions (50 in Africa) in August 2009 Embedded learning and evaluation project: 1. Annual monitoring of each award through completion of bespoke report template 2. Evaluation across all research capacity strengthening awards at key points African Institutions Initiative

10. ⁺ Budget built into the development of the Initiative ⁺ Monitoring and evaluation embedded from the start ⁺ Worked with consortia to develop the monitoring ⁺ Developed framework to collect consistent data across consortia African Institutions Initiative − Lack of routinely collected available data − Overburden/proportionality of requirements − Resourcing/capacity: consortia, WT and RAND − Managing relationships and expectations

11. prospective system to track the career choices & progression of cohorts of individuals over time Wellcome Trust Career Tracker annual online survey; ‘panel’ approach trends in career progression across cohorts understanding the influences on career choices developing individuals & building research capacity is vital making sure our funding mechanisms work for the future of science

12. Wellcome Trust Career Tracker ⁺Three year’s worth of data and starting to see trends and differences across cohorts (wave 4 underway) ⁺Achieve very good response rates circa 80% ⁺Commissioning up follow-up qual work ⁺Rolling out for other cohorts ₋Resource needed to set-up and keep response high ₋High level – to keep q’aire short and allow benchmarking ₋How long will people keep responding?