12630 Statistics for Authors Canadian Journal of Anesthesia Journal canadien d’anesthésie Winterlude Anesthesia Symposium February 1, 2015 Ottawa, Ontario

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Short version of this session Hire an experienced PhD biostatistician … before you submit to ethics … understand your data, but don’t touch it … statistician writes statistical methods (results). Thank you, let’s go watch the SuperBowl.

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Disclosure No financial COI Not, definitely not, a statistician Not particularly good at math MSc Epidemiology Deputy Editor-in-Chief, Canadian Journal of Anesthesia I do a lot of peer review

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Objectives I will not tell you how to do statistics I will not show you formulae Discuss common statistical problems identified at peer review Design Analysis Reporting Links and take home points on #wntrane15

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Studies are small Effect sizes are small Greater number of less predetermined statistical tests Flexibility in design, outcomes, and analysis Conflict of interest Multiple teams competing

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 What is the research question? P = Population I = Intervention C = Comparator O = Outcome

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q1. What percent of trials change their primary outcome between registration and publication? A.None B.3% C.10% D.30%

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 TrialsN (%) Identified323 Registered147 (46) Different primary outcome 46 (31) Change statistically significant 19 (41) Things change … JAMA. 2009;302(9):

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q1. What percent of trials change their primary outcome between registration and publication? A.None B.3% C.10% D.30%

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q2. What is a meaningful change in pain score? A.1 point B.10% C.2 points D.30%

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Does it feel better? 2,700 patients in 10 pregabalin-neuropathy trials 5 to 12 weeks of therapy NRS pain cw 7 point “improved” scale points or -27.9% decrease 134 patients, fentanyl cancer-pain breakthrough Measurements every 15 minutes for 1 hour Success = not requiring additional medication at 30 minutes - 2 points or <33% decrease Farrar JT Pain 2001;94;149–158 Farrar JT. J Pain Sympt Manage 2003;25:406-11

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q2. What is a meaningful change in pain score? A.1 point B.10% C.2 points D.30%

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q3. What’s wrong with this sample size estimate? In this three-group trial the authors state “… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group”

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Required Elements for Sample Size Estimate MeanProportion Alpha (type 1) error Beta (1-power) error Mean interventionRate intervention Mean controlRate control Standard deviation

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q3. Identify errors in this sample size estimate In this three-group trial the authors state “… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group” CategoryError Central tendencyMean pain in target population DispersionSD pain in target population

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q4. How many comparisons in 3 group trial? ACB

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q5. Are we any good at sample size? Two years worth of top medicine journals NEJM, JAMA, PLoS Med, Lancet, BMJ, Ann Int Med 215 citations with median sample size of (53%) reported all elements required 146 (68%) assumptions were <30% off observed result 73 (34%) estimates were complete and accurate 96 (45%) were registered with a sample size estimate 46 (21%) registration matched report BMJ 2009;338:b1732

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Treatment N=30 Control N=30 P Value 48 hr morphine27 (15)32 (16)0.12 NRS pain2 (3)3(3)0.23 BPI function4 (5)4 (6)0.78 SF-MPQ (3.6)3.6 (3.1)0.78 Continuous3.2 (2.5)4.0 (2.0)0.04* Intermittent3.3 (2.6)2.8 (3.0)0.23 Neuropathic2.2 (2.9)2.1 (2.4)0.87 Affective2.6 (2.8)2.4 (2.4)0.75 Q6. What’s wrong with this table? Data presented as mean (SD). P for unpaired T-test indicated

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Treatment N=30 Control N=30 P Value PACU7 (8)8 (10) hr6 (6)8 (10) hr2 (5)6 (8)0.04* 24 hr6 (8)6 (9) hr6 (10)4 (10)0.32 Total27 (15)32 (16)0.12 Q7. Or this one? Data presented as mean (SD). P for unpaired T-test indicated

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Answer Qs 6 and 7 = Multiplicity Alpha errors accumulate FWER = 1 – (1-alpha) number of tests FWER = 1 – (1-0.05) 8 = = 33.7% Multiple primary outcomes Multiple related outcomes Repeated measures Multiple treatment groups Interim analyses The more cuts at the data the more like you are to make a type 1 error.

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 P-hacking Coined by Joseph Simmons, Leif Nelson, and Uri Simonsohn Generally refers to repeated analysis of data until P < 0.05 appears Complicated debate, Bayes’ theorem, false discovery rates Bottom line. Analyze only what you need In the way your protocol described it

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q8. Do we favor positive results? Kühberger, Fritz, and Schendl. PLoSOne 2014;9(9):e

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q8. Do we favor positive results?

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 P values by XKCD Or if all else fails, use “significant at P>0.05” and hope no one notices.

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Q9. Which of the following is “better?” A.Morphine consumption reduced cw placebo (P 0.017) B.33% reduction in morphine consumption cw placebo (P 0.017) C.Morphine consumption 18mg (7) v 12mg (6) (P 0.017) D.Mean difference morphine consumption 6mg (95% CI 2-9) P values conflate statistical and clinical significance Must report measure of effect Absolute measure preferred to relative. Mean difference with 95% confidence limits Absolute risk reduction or NNT

Canadian Journal of Anesthesia Journal canadien d‘anesthésie springer.com/12630 Registration Replication Open data Standardized outcomes More stringent statistical approach