NC3Rs resources to improve the design of animal experiments Dr Nathalie Percie du Sert BBSRC STARS Course Tuesday 11 April 2017

The NC3Rs (UK National Centre for the 3Rs) Lead on the discovery and application of new technologies and approaches to replace, reduce and refine the use of animals for scientific purposes (the 3Rs) Visit our website: www.nc3rs.org.uk @NC3Rs National Centre for the 3Rs Work with research funders, journals, academia, industry and regulators Activities divided between: Research funding Centre-led science programmes

NC3Rs resources The ARRIVE guidelines and the Experimental Design Assistant

Background Quality of published animal research Experimental design Only 12% of publications report randomisation and 14% report blinding Sample size justification – missing in 100% Reporting of studies Animal characteristics – missing in 25% Only 59% stated the study hypothesis, number and characteristics of animals used Statistical analysis Only 70% of publications fully described the statistical methods and presented the results with a measure of variability Survey reviewed 271 publications and identified key areas for improvement Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, et al. (2009). Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 4(11): e7824.

Experimental design – internal validity Consider threats which might compromise the validity of the experiment, any opportunities for the investigator to influence: animal selection conduct of the experiment assessment of outcome which outcomes are reported Measures used to reduce validity threats include: Random allocation to treatment groups Allocation concealment Blinding during outcome assessment Inclusion/exclusion criteria

Experimental design – randomisation Method is important – haphazard is not random Use a validated procedure (e.g. computer generated, throw a dice, flip a coin) Randomisation is crucial for two reasons: Minimise selection bias e.g. haphazard selection may results in slowest mice allocated to the same group Key assumption of the statistical analysis Different groups should be drawn from the same background population using random sampling

Experimental design – randomisation 347 responses Is 17 the “most random” number? Pick a number between 1 and 20 http://scienceblogs.com/cognitivedaily/2007/02/05/is-17-the-most-random-number/

Experimental design – blinding 12 students Maze-bright and maze-dull rats Elevated T-maze, dark arm reinforced Rats had been labelled bright or dull randomly Only difference was in the minds of the investigators! Rosenthal R, Fode KL (1963). The effect of experimenter bias on the performance of the albino rat. Behavioral Science 8(3): 183-189.

Improving the reporting of in vivo research The ARRIVE guidelines The ARRIVE guidelines were developed to improve the reporting of biomedical research using animals. Checklist of 20 items, containing key information necessary to describe a study comprehensively and transparently. The guidelines include: Information which relates to internal validity Information which would allow a study to be repeated Information about the context and scientific relevance of the study https://www.nc3rs.org.uk/arrive-guidelines

The EDA Experimental Design Assistant The EDA was developed to improve the design of animal experiments Web-based tool Aimed at in vivo researchers Developed as a collaboration between: In vivo researchers Statisticians Academia and industry Software designers specialised in artificial intelligence Road tested by researchers and statisticians https://eda.nc3rs.org.uk

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Effect of drug A on plasma glucose levels Animals characteristics: diabetic mice Experimental unit:: mouse Vehicle Drug Measurement: Plasma glucose Outcome Measure: Glucose levels Analysis Group 2 Group 1 Pool of animals Allocation: Complete randomisation Pharmacological intervention 1 Pharmacological intervention 2 Independent variable of interest : Drug A

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Practical steps Analysis Experiment Effect of drug A on plasma glucose levels Animals characteristics: diabetic mice Experimental unit:: mouse Vehicle Drug Measurement: Plasma glucose Outcome Measure: Glucose levels Analysis Group 2 Group 1 Pool of animals Allocation: Complete randomisation Pharmacological intervention 1 Pharmacological intervention 2 Independent variable of interest : Drug A

The EDA diagram Examples Templates Feedback from the critique

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Feedback and advice on your experimental plan Dedicated support for randomisation, blinding and sample size calculation Practical information to improve knowledge of experimental design Improved transparency of the experimental plan, allowing more efficient communication

Feedback and advice from the EDA Dataset of rules triggers prompts based on the EDA diagram Feedback provided: Diagram structure Ask to provide more information Point out inconsistencies Prompt you to consider things that are not on the diagram Highlight implications of some of the choices made Provide recommendation for analysis Rule set to be expanded over time Provide more feedback Enable the system to detect more subtle issues Increasingly specific feedback

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Feedback and advice on your experimental plan Dedicated support for randomisation, blinding and sample size calculation Practical information to improve knowledge of experimental design Improves transparency of the experimental plan and helps communication

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Feedback and advice on your experimental plan Dedicated support for randomisation, blinding and sample size calculation Practical information to improve knowledge of experimental design Improved transparency of the experimental plan, allowing more efficient communication

The EDA offers: The ability to build a stepwise visual representation of an experiment – the EDA diagram Feedback and advice on your experimental plan Dedicated support for randomisation, blinding and sample size calculation Practical information to improve knowledge of experimental design Improved transparency of the experimental plan, allowing more efficient communication

The EDA workflow

Objectives Improve the reliability of published results Promote better understanding of experimental design, raise awareness about issues Facilitate peer review/assessment of the experimental plans with an explicit description Transparency Pre-registration Promote more careful consideration of the experimental plans Spend time planning Diagram facilitate discussion

EDA uptake ~2500 accounts on the system with 30 diagrams created per week Recommended by UK funders

Acknowledgments https://eda.nc3rs.org.uk www.nc3rs.org.uk/ARRIVE NC3RS working group Prof Clare Stanford (Chair), UCL Dr Simon Bate, GlaxoSmithKline Dr Manuel Berdoy, University of Oxford Dr Robin Clark, Huntingdon Life Sciences Prof Innes Cuthill, University of Bristol Dr Derek Fry, University of Manchester Dr Natasha Karp, Wellcome Trust Sanger Institute Prof Malcolm Macleod, University of Edinburgh Dr Lawrence Moon, King’s College London Dr Richard Preziosi, University of Manchester https://eda.nc3rs.org.uk www.nc3rs.org.uk/ARRIVE Certus Technology Dr Brian Lings Mr Ian Bamsey Alpha testers Beta testers