University of Sydney Statistics 101: Power, p-values and ………... publications. Dr. Gordon S Doig, Senior Lecturer in Intensive Care, Northern Clinical School

Analysis 101: The basic tests t-test paired t-test Wilcoxon Rank Sum test (Mann-Whitney U test) Wilcoxon Signed Rank Sum test Kolmogorov-Smirnov (one and two sample test) Chi-square test Fisher’s Exact test ANOVA Kruskal-Wallis rank test repeated measures ANOVA

Why do we need statistics??? When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions: I) II)

Why do we need statistics??? When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions: I) we claim to have found an important treatment effect when in reality there is no treatment effect. II) we claim that no treatment effect exists when in reality there is an important treatment effect.

Why do we need statistics??? Some important definitions: What is a p-value? What is power?

Why do we need statistics??? Some important definitions: What is a p-value? What is power? P-value: The probability that the difference we observed could be due to chance alone.

Why do we need statistics??? Some important definitions: What is a p-value? What is power? P-value: The probability that the difference we observed could be due to chance alone. Power: The probability that if there is a real difference, our experiment will find it.

Why do we need statistics??? When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions: I) we can claim to have found an important treatment effect when in reality there is no treatment effect. II) we can claim that no treatment effect exists when in reality there is an important treatment effect P-value: The probability that the difference we observed could be due to chance alone. Power: The probability that if there is a real difference, our experiment will find it.

Sample size calculations: The use of Power Every experiment should start with a sample size calculation. Having adequate power protects us from Type II errors.

Sample size calculations: The use of Power Every experiment should start with a sample size calculation. Having adequate power protects us from Type II errors. Forces us to consider a primary outcome for our experiment. primary outcomes can be continuous, categorical (interval, ordered, unordered), dichotomous

Sample size calculations: The use of Power Every experiment should start with a sample size calculation. Having adequate power protects us from Type II errors. Forces us to consider a primary outcome for our experiment. primary outcomes can be continuous, categorical (interval, ordered, unordered), dichotomous Should consider issues of design in order to simplify analysis.

Analysis 101: The use of P??? Selection of appropriate study design / analytic technique: protects from Type I errors. is driven by driven by a combination of study outcome and study design.

Analysis 101: Basics of experimental design 1) Before and after trial physiological parameter/outcome measured intervention delivered physiological parameter/outcome measured again compare measurement before with measurement after, usually in same subject

Analysis 101: Basics of experimental design 1) Before and after trial physiological parameter/outcome measured intervention delivered physiological parameter/outcome measured again compare measurement before with measurement after, usually in same subject 2) Comparison between two groups subjects are randomly assigned to one of two groups one group receives intervention compare outcome between two groups after intervention

Analysis 101: Basics of experimental design 1) Before and after trial physiological parameter/outcome measured intervention delivered physiological parameter/outcome measured again compare measurement before with measurement after, usually in same subject 2) Comparison between two groups subjects are randomly assigned to one of two groups one group receives intervention compare outcome between two groups after intervention 3) Comparison between more than two groups as above but subjects are assigned to more than two groups could compare 3 different drugs or 3 different doses

Analysis 101: Outcome identification Primary outcomes can be 1) continuous, 2) categorical (interval, ordered, unordered), 3) dichotomous 1) Continuous outcomes: most physiological parameters (Hb, pressures, biochemistry) usually involves a direct measurement often Normally distributed

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category length of stay, age, time to event, some scoring systems may be Normally distributed

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category length of stay, age, time to event, some scoring systems may be Normally distributed b) ordered unequal unit change between each ordered category

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category length of stay, age, time to event, some scoring systems may be Normally distributed b) ordered unequal unit change between each ordered category most scoring systems, tumor stage or grade, low-medium- high not usually Normally distributed

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category length of stay, age, time to event, some scoring systems may be Normally distributed b) ordered unequal unit change between each ordered category most scoring systems, tumor stage or grade, low-medium- high not usually Normally distributed c) unordered no sequential order to categories

Analysis 101: Outcome identification 2) Categorical outcomes: a) interval equal unit change between each ordered category length of stay, age, time to event, some scoring systems may be Normally distributed b) ordered unequal unit change between each ordered category most scoring systems, tumor stage or grade, low-medium- high not usually Normally distributed c) unordered no sequential order to categories type of tumor, location, diagnosis re-think outcome selection!!!!

Analysis 101: Outcome identification 3) Dichotomous outcomes: only two possible outcome states tumor / no tumor dead / alive follows Binomial distribution

Analysis 101: The basic tests t-test paired t-test Wilcoxon Rank Sum test (Mann-Whitney U test) Wilcoxon Signed Rank Sum test Kolmogorov-Smirnov (one and two sample test) Chi-square test Fisher’s Exact test ANOVA Kruskal-Wallis rank test repeated measures ANOVA

Analysis 101: Design and Analysis 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis Step 1: Determine if outcome is Normally distributed plot histogram with density function line 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis Step 1: Determine if outcome is Normally distributed plot histogram with density function line could ‘formally’ test using Wilkes-Shapiro statistic 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis Step 1: Determine if outcome is Normally distributed plot histogram with density function line could ‘formally’ test using Wilkes-Shapiro statistic 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis paired t-testWilcoxon Signed Rank Sum Test 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis paired t-testWilcoxon Signed Rank Sum Test NB - if ordered categorical outcome, use one sample Kolmogorov- Smirnov test 1) Before and after trial (same subjects, continuous and interval )

Analysis 101: Design and Analysis 2) Comparison between two groups(continuous and interval)

Analysis 101: Design and Analysis 2) Comparison between two groups Step 1: Determine if outcome is Normally distributed plot histogram (use all data) with density function line could ‘formally’ test using Wilkes-Shapiro statistic (continuous and interval)

Analysis 101: Design and Analysis 2) Comparison between two groups Step 1: Determine if outcome is Normally distributed plot histogram (use all data) with density function line could ‘formally’ test using Wilkes-Shapiro statistic (continuous and interval)

Analysis 101: Design and Analysis 2) Comparison between two groups(continuous and interval)

Analysis 101: Design and Analysis 2) Comparison between two groups t-testWilcoxon Rank Sum test (continuous and interval)

Analysis 101: Design and Analysis 2) Comparison between two groups t-testWilcoxon Rank Sum test NB - if ordered categorical outcome, use two sample Kolmogorov- Smirnov test (continuous and interval)

Analysis 101: Design and Analysis 3) Comparison between more than two groups

Analysis 101: Design and Analysis 3) Comparison between more than two groups Step 1: Determine if outcome is Normally distributed plot histogram (use all data) with density function line could ‘formally’ test using Wilkes-Shapiro statistic

Analysis 101: Design and Analysis 3) Comparison between more than two groups

Analysis 101: Design and Analysis 3) Comparison between more than two groups ANOVAKruskal-Wallis rank test

Analysis 101: Design and Analysis 3) Comparison between more than two groups ANOVAKruskal-Wallis rank test NB - could transform (calculate the log or ln) each outcome value and redo histogram…. if transformed values are Normally distributed, can now use ‘more powerful’ ANOVA (or t-test if 2 samples).

Analysis 101: Dichotomous outcomes 1) Before and after trial rate before intervention compared to rate after intervention McNemer’s chi-square

Analysis 101: Dichotomous outcomes 1) Before and after trial rate before intervention compared to rate after intervention McNemer’s chi-square 2) Comparison between two groups create 2x2 table, calculate rate for each Group DeadAlive Group A % mortality Group B % mortality compare using chi-square test

Analysis 101: Dichotomous outcomes 1) Before and after trial rate before intervention compared to rate after intervention McNemer’s chi-square 2) Comparison between two groups create 2x2 table, calculate rate for each Group DeadAlive Group A % mortality Group B % mortality compare using chi-square test NB - if any one cell contains < 5 counts, use Fisher’s Exact test

Analysis 101: Dichotomous outcomes 1) Before and after trial rate before intervention compared to rate after intervention McNemer’s chi-square 2) Comparison between two groups create 2x2 table, calculate rate for each Group DeadAlive Group A % mortality Group B % mortality compare using chi-square test NB - if any one cell contains < 5 counts, use Fisher’s Exact test 3) Comparison between more than two groups undertake a series of comparisons via 2x2 tables as above

Analysis 101: Special considerations Transformations: Sometimes its possible to ‘transform’ a long tailed distribution to a normal distribution. Calculate the log or ln of each outcome value and redo histogram. Allows us to apply ‘more powerful’ tests based on assumption of Normality (paired t-test, t-test, ANOVA). Try non-parametric test first <- fewer assumptions!!!!

Analysis 101: Special considerations The t-test has 3 basic, fundamental underlying assumptions: 1) Outcomes are Normally distributed test assumptions of Normality use non-parametric tests

Analysis 101: Special considerations The t-test has 3 basic, fundamental underlying assumptions: 1) Outcomes are Normally distributed test assumptions of Normality use non-parametric tests 2) Outcomes are independent if outcomes are from same subjects, use paired t-test

Analysis 101: Special considerations The t-test has 3 basic, fundamental underlying assumptions: 1) Outcomes are Normally distributed test assumptions of Normality use non-parametric tests 2) Outcomes are independent if outcomes are from same subjects, use paired t-test 3) The variance of each group is similar stats package should formally test equality of variances different p-values for each condition

Analysis 101: Summary t-test (two groups, Normally distributed) paired t-test (before/after, Normally distributed) Wilcoxon Rank Sum test (two groups, non-parametric) Wilcoxon Signed Rank Sum test (before/after, non-parametric) Kolmogorov-Smirnov (before/after, two groups, ordered categorical) Chi-square test (dichotomous outcome) Fisher’s Exact test (dichotomous outcome, any cell size < 5) ANOVA (more than two groups, Normally distributed) Kruskal-Wallis rank test (more than two groups, non-parametric) repeated measures ANOVA