STUDY DESIGN: PILOT STUDIES Charles Flexner, MD Johns Hopkins University

Pilot Studies: A Case Presentation

What is a Pilot Study???

n Pilot Study n Developmental Study n Feasibility Study n Phase I Study n Small Exploratory Clinical Trial n Hypothesis Generating Study u As opposed to hypothesis-testing n Etc. Semantics

n The initial study examining a new method or treatment. F -- NCI Dictionary of Cancer Terms n A small scale preliminary study conducted before the main research in order to check the feasibility or to improve the design of the research. F -- Wikipedia! What is a Pilot Study?

n “In a comprehensive literature search using Medline and the Web of Science we could find no formal methodological guidance as to what constitutes a pilot study.” u --- GA Lancaster, S Dodd, PR Williamson. Design and Analysis of Pilot Studies: Recommendations for Good Practice. Journal of Evaluation in Clinical Practice, 2004; 10(2): What is a Pilot Study?

--- GA Lancaster, et al. Journal of Evaluation in Clinical Practice, 2004; 10(2):

What question is usually being asked in a pilot study? The Bottom Line

What question is usually being asked in a pilot study? n “How do I get started?” n “Is this technique / intervention / data mining exercise feasible?” n “Do I need preliminary data to prove that my sample size is correct?” n “How much data do I need to justify doing further studies related to this hypothesis?” The Bottom Line

Pilot Studies: Some published examples

n Educated uncertainty n Feasibility concerns n Sample size concerns n Critical developmental needs u Assay development and/or standardization u New study instrument u New device or technique for endpoint measurement Good reasons to do a “Pilot Study”

n Inadequate literature review n Need to generate hypotheses n Running out of: u Time u Money u Patients u Patience n Laziness Bad reasons to do a “Pilot Study”

n Test integrity & feasibility u Recruitment & consent u Intervention (e.g. tolerance, compliance, retention) u Data collection (e.g. forms, interface, time) u Equipment u Other procedures (e.g. randomization) n Refine methods and procedures n Confirm or revise sample size estimates PILOT STUDIES: “Pretesting”

n Recruitment & consent: u Get the types of participants that we think we will get? u Any important segments of the target population being left out? u People turn down the opportunity to participate in our study? (what proportion? able to meet the sample size requirement in time? recruitment pool large enough? expand the inclusion criteria or go multi-center?) u Is it obvious who meets and who does not meet the eligibility requirements? u Can this be learned without a formal feasibility study? PILOT STUDIES: “Pretesting”

n Compliance and Retention: u Acceptability / tolerability of intervention u Participants fail to comply with protocol requirements? (what proportion? need to modify protocol? revise analytic plan?) u Participants fail to finish our study? (what proportion? reduce participant burden? add run-in? increase sample size?) PILOT STUDIES: “Pretesting”

n Data Collection: u Are all the important/required data items collected? (run through the analytic plan) u Is there enough room on the data collection form for all of the data you receive? u Who will be recording the data? (standardized training? standardized equipment & calibration? standardized procedures? blinding/masking procedures?) u Any problems entering collected data? u Are the data collection instruments validated and reliable in the target population? PILOT STUDIES: “Pretesting”

“Murphy’s Law: anything that can go wrong will go wrong. “The reason to run a pilot study is to ensure that the things that do go wrong, go wrong during the pilot study so we can fix them before we start the full study!” – Nae-Yuh Wang

n Hard Endpoints n Soft Endpoints n Clinical Endpoints n Surrogate Endpoints n Biomarkers n Genes (SNP’s, Haplotypes, GWAS) PILOT STUDIES: Endpoints

Don’t forget! The value of n-of-1 studies and self-experimentation

n Importance of adhering to the classical experimental method u Hypothesis testing u Sample size calculation u Proper endpoint selection u Proper study population n Give preference to publishable endeavors! The Bottom Line

Do it right the first time!!!” - Scott Zeger, PhD