Pilot studies Karla Hemming The University of Birmingham

No consensus on the definition Pilot study: Mini version of the full trial Tests protocol Feasibility study: Tests individual components Refines / develops intervention 2 Pilot or feasibility study?

70% of publically funded trials fail to recruit required number of participants NIHR increasingly require demonstration of proof of principal that the TRIAL will work (not the intervention) 3 Why have a pilot study?

Not around estimating effect size or CI of effect size Why? Pilot studies are small Any estimate of effect will be highly uncertain Mean estimate tells us nothing! 4 What pilot studies are not about

Show proof of principal of intervention (?) Test out data collection methods Acceptability of questionnaires Inform outcome Only insofar as what is important to the patient Estimate important parameters to inform sample size calculation (not effect size!) 5 Objectives of a pilot study

Need a well defined surrogate outcome That surrogate outcome would have to have a very clear link to important clinical outcome Example – weight management programme Important clinical outcome BMI Show adherence improved in a pilot? Difficulty – to show adherence improved might need as large a sample as for BMI 6 Proof of principal

Examples include but are not limited to: Recruitment rate (informs how long the trial will have to last for) Consent rate (informs how amenable patients are to be randomised) Retention rate (inform drop out rate) Continuous outcomes – SD Binary outcomes – underlying event rate 7 Important parameters to estimate

Sample size should be justified Depends on objective 8 Sample size justification

Browne 1995 Sample size of 30 is reasonable to give an estimate of the SD BUT! Don’t just use mean estimate of SD in subsequent power calculations – use upper 95 percentile value 9 Objective: estimate SD

Show this rate is above some value Or within some range (CI) Called a precision based approach ) 10 Objective: estimate a process rate

Want to show retention rate (p) is above some value (c) Retention should not differ between arms (blinded trial) – so include both arms in calculation Precision based approach Show retention rate above c% Need an estimate of actual retention rate p% Specify significance or width of CI (precision) 11 Example

Show retention rate above 60% Estimate actual retention rate 80% Recruit SS of 35, number retained 28 (80%) 95% CI would be: 63% to 92% Great! Showed retention rate above 60% and can proceed to definitive trial 12 Example cont…

What if retention rate only 70% That is, recruit 35 But, only 24 are retained 95% CI would be 51% to 83% Oh no… haven't shown retention rate above 60% 13 Example cont…

Method was motivated around CI or precision based approach (common in pilot studies) But, the power of a SS of 35 to show that the retention rate is above 60% is only 50% (assuming event rate observed is 80%) 14 Power of this sample size

Precision methods (CI) crucially depend on actual observed rate Associated power of this approach is much lower than 80% So, if you use the precision based approach do not incorporate stringent criteria to follow to definitive trial 15 Precision based methods

Application viewed unfavourably if geared around effect sizes and hypothesis testing Rather, should be focused around testing TRIAL protocol (or components) Sample size needs to be justified Don’t have stringent stop go criteria for full trial 16 Recommendations

More than one objective – take the larger SS Internal pilots Full trial fundable? If full trial not likely to be fundable by NIHR, RfPB unlikely to fund 17 Other things…