Nicola Totton Bangor University Review of pilot and feasibility studies from a registry website; an overview of recent practice. Nicola Totton Bangor University

Introduction Pilot and feasibility studies are small studies used as a precursor to completing a full clinical trial Method for evaluating the trial suitability and testing study components in practice Aim to assess recruitment and retention rates Pilot and feasibility work recommended by the MRC1 and several researchers2-6 Launch of the Journal of Pilot and Feasibility Studies in January 2015 Developing and evaluating complex interventions: new guidance. MRC, 2006. A tutorial on pilot studies: the what, why and how. Thabane L et al (2010). BMC Medical Research Methodology. Design and analysis of pilot studies: recommendation for good practice. Lancaster GA, Dodd S & Williamson PR. (2002). Journal of Evaluation in Clinical Practice. From pilot studies to confirmatory studies. N. Duan, 2013. Shanghai Archives of Psychiatry. Pilot and feasibility studies come of age! G Lancaster, 2015. Pilot and Feasibility Studies. The importance of pilot studies. ER van Teijlingen & V Hundley. Social Research Updated, University of Surrey.

What are the differences? The terms are often used interchangeably A framework8 was published in 2016 to define Feasibility studies are exploratory and an overarching term when assessing whether a future study can be done Pilot studies are a mini replication of a future study Pilot studies are a specific subset of feasibility studies Feasibility Studies Pilot Studies In practice, these terms are often used interchangeably to describe a small scale trial and there is confusion with the difference between the two. In 2016, a framework was published which aims to clearly define what is a pilot and what is a feasibility study. This suggested that the terms were not mutually exclusive and that a pilot study can be considered as a specific type of feasibility study. 8. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. Eldridge SM, Lancaster GA et al, 2016.

Aims Prevalence: Evaluate the prevalence of pilot and feasibility studies over time Characteristics: Assess key characteristics and whether these differ between pilot and feasibility studies Issues: Identify issues currently present within pilot and feasibility research

Methods The ISRCTN website is an international clinical trials registry A web scraping methodology was adopted using R Search terms “pilot” and “feasibility” were used on the public title field Pilot term returned 474 records and feasibility returned 215 records In total 664 unique studies were identified Andrew Brand (NWORTH) has used a similar methodology to create an online tool for easily evaluating the characteristics of registered pilot and feasibility studies. A poster describing the system (An Online Tool for Exploring Recruitment Achievability for a Feasibility and Pilot Studies in the UK) can be found on poster board 12, Hall 2A.

Methods Prevalence: The proportion of pilot and feasibility trials from all registered studies assessed per year Characteristics: Data such as target sample size, type of study, start and end date etc. were extracted into an Excel file Subset of trials evaluated to in more detail and compared to 2007-8 review to assess changes over time Issues: Identify any issues arising from previous studies found from the subset

Results - Prevalence 69% pilot studies 30% feasibility studies 1% both a pilot and feasibility study

Results - Prevalence The overall proportion remains consistent but number of feasibility studies has increased to match pilot studies Whilst there is some fluctuation, the overall proportion of trials labelled as pilot or feasibility generally remains consistent over time, possibly with a slightly increase as shown by the dark red line. The red line shows pilot studies were initially more common, however in recent years feasibility studies, shown by the orange line, have increased in popularity and are now on a par with pilot studies.

Results - Characteristics Median target participants per trial: Overall - 50 (range: 5-130,000) Pilot studies - 44 (range: 5-130,000) Feasibility studies - 60 (range: 6-1,600) Median time period for the trial: Overall - 18 months (range: 0-126) Pilot studies - 17 months (range: 0-126) Feasibility studies - 18.5 months (range: 0-107)

Results - Characteristics Condition category: Mental and behavioural disorder studies (20%) Cancer studies (13%) Study type: Interventional (96%) Randomised controlled trials (85%) Setting: Hospital (45%) Recruitment: Single country (97%)

Results – Characteristics Trial Component Included? ISRCTN Subset (2011) Overall (n=31) Pilot (n=22) Feasibility (n=9) Sample size calculation 32% 25% Randomisation 74% 73% 75% Control group 77% 63% Blinding 26% 36% 0% Hypothesis testing 33% 38% Further study suggested 78% 83% 67% 2007-8 Review13 (n=23) 35% 62% 69% 19% 81% 13. What is a pilot or feasibility study? A review of current practice and editorial policy. Arian et al, 2010. BMC Medical Research Methodology.

Results – Issues Publication bias Pilot and feasibility terms assigned incorrectly in 29% of cases Unclear motivation Unclear aims of the study Inappropriate efficacy testing

Conclusions Pilot and feasibility studies are shorter than full trials with less participants Slight differences within the characteristics between pilot and feasibility studies Studies sometimes mislabelled and have unclear motivation Aims do not line up with the intended design Results often not being published

Thank you Any Questions?