1. Outcomes Research Chapter 5 Cummings 5 th ed. Darshni Vira

2. AKA clinical epidemiology AKA clinical epidemiology Study of the effectiveness of treatment in a nonrandomized, real-world setting (observational data) Outcome measures - survival, costs, physiologic measures, QOL

3. Study Outline Pt presents at baseline with a condition Receives treatment for that condition Experiences a response to treatment

4. Bias and Confounders Bias - “Compared components are not sufficiently similar” Selection bias Treatment bias Confounders – “V Confounders – “Variable thought to cause an outcome is actually not responsible because of the unseen effects of another variable age, gender, ethnicity, race, comorbidities

5. Assessment of Baseline Condition Definition of disease Definition of disease Inclusion criteria Inclusion criteria Disease severity Disease severity TNM TNM Sinusitis – Lund-Mackay, Harvard, etc  reproducible results Sinusitis – Lund-Mackay, Harvard, etc  reproducible results Comorbidity Comorbidity Adult Comorbidity Evaluation 27 (ACE-27) is a validated instrument for evaluating comorbidity in cancer patients

6. Assessment of Treatment Control Groups Control Groups

7. Assessment of Outcomes Efficacy Efficacy Health intervention, in a controlled environment, achieves better outcomes than does placebo Effectiveness Retains its value under usual clinical circumstances

8. Study Design DesignAdvantagesDisadvantages Level of Evidence Randomized clinical trial (RCT) Only design to prove causation Unbiased distribution of confounding Expensive and complex Typically targets efficacy 1, if high-quality RCT 2, if low-quality RCT Observational (cohort) study Cheaper than RCT Clear temporal directionality from treatment to outcome Difficult to find suitable controls Confounding 2, with control group 4, if no control group Case-control studyCheaper than cohort study Efficient study of rare diseases or delayed outcomes Must rely on retrospective data Directionality between exposure and outcome unclear3 Case seriesCheap and simpleNo control group No causal link between treatment and outcome4 Expert opinionn/an/a5

9. Grade of Recommendation (EBM) Level of Evidence A1 B 2 or 3 C4 D5

10. Measurement of Clinical Outcomes Psychometric Validation (questionnaires) Psychometric Validation (questionnaires) Reliability Reliability Validation Validation Responsiveness Responsiveness Burden Burden

11. Categories of Outcomes Health Status Health Status Individual’s physical, emotional, and social capabilities and limitations Function How well an individual is able to perform important roles, tasks, or activities QOL Central focus is on the value that individuals place on their health status and function

12. Examples of Outcome Measures Medical Outcomes Study Short Form-36 (SF-36) European Organization for Research and Treatment of Cancer Quality of Life Questionnaire (EORTC-HN35) Hearing Handicap Inventory in the Elderly (HHIE) Sinonasal Outcome Test (SNOT-20) Child Health Questionnaire (CHQ) Voice Handicap Index Functional Outcomes of Sleep Questionnaire (FOSQ)

13. Interpreting Medical Data Chapter 6 Cummings 5 th ed.

14. Habits of a Highly Effective Data User 1. Check quality before quantity 2. Describe before you analyze 3. Accept the uncertainty of all data 4. Measure error with the right statistical test 5. Put clinical importance before statistical significance 6. Seek the sample source 7. View science as a cumulative process

15. 1. Check Quality before Quantity Experimental vs observational study Experimental vs observational study Bias Bias Confounders Confounders Control group Control group Placebo response Placebo response Prospective studies measure incidence (new events) whereas retrospective studies measure prevalence (existing events)

16. 2. Describe Before You Analyze Begins by defining the measurement scale that best suits the observations Categorical (qualitative) Numerical (quantitative) Bell-shaped curve with standard deviation Median Survival curve

17. Categorical ScaleDefinitionExample Dichotomoustwo mutually exclusive categories Breastfeeding (yes/no), sex (male/female) Nominalunordered qualitative categories Race, religion, country of origin Ordinalordered qualitative categories, but with no natural (numerical) distance between their possible values Hearing loss (none, mild, moderate), patient satisfaction (low, medium, high), age group

18. Odds ratio with retrospective review Odds ratio with retrospective review Relative risk with prospective review Relative risk with prospective review Rate difference with prospective trials Rate difference with prospective trials Correlation coefficient with ordinal or numerical data Correlation coefficient with ordinal or numerical data Coefficient (r) from 0 to 0.25 indicates little or no relationship, from 0.25 to 0.49 a fair relationship, from 0.50 to 0.74 a moderate to good relationship, and greater than 0.75 a good to excellent relationship. A perfect linear relationship would yield a coefficient of 1.00

19. 3. Accept the Uncertainty in All Data Precision (repeatability) Should be reported with a 95% confidence interval Precision may be increased by using a more reproducible measure, by increasing the number of observations (sample size), or by decreasing the variability among the observations Accuracy measures nearness to the truth measured in an unbiased manner and reflect what is truly purported to be measured

20. 4. Measure Error with the Right Statistical Test All statistical tests measure error All statistical tests measure error Choosing the right test is determined by Choosing the right test is determined by (1) whether the observations come from independent or related samples, (2) whether the purpose is to compare groups or to associate an outcome with one or more predictor variables, and (3) the measurement scale of the variables

21. Null hypothesis Results observed in a study, experiment, or test that are no different from what might have occurred due to chance alone Statistical testProcedure used to reject or accept a null hypothesis Type I (alpha) errorRejecting a true null hypothesis (false- positive error); declaring that a difference exists when in fact it does not P valueProbability of making a type I error; P <.05 indicates a statistically significant result that is unlikely to be caused by chance Type II (beta) errorAccepting a false null hypothesis (false- negative error); declaring that a difference does not exist when in fact it does PowerProbability that the null hypothesis will be rejected if it is indeed false; mathematically, power is 1.00 minus type II error

22. 5. Putting Clinical Importance Before Statistical Significance The next logical question after “Is there a difference?” (statistical significance) is “How big a difference is there?” (clinical importance) Effect size reflects the magnitude of difference between groups Measured by correlation coefficient Confidence intervals (CI) are more appropriate measures of clinical importance than P values, because they reflect both magnitude and precision If “significant” results, the lower limit of the 95% CI should be scrutinized; a value of minimal clinical importance suggests low precision (inadequate sample size) If “nonsignificant” results, the upper limit of the 95% CI should be scrutinized; a value indicating a potentially important clinical effect suggests low statistical power (false-negative finding)

23. 6. Seek the Sample Source A statistical test is valid only when the study sample is random and representative Identifying the sampling method and selection criteria (inclusion and exclusion criteria) that were applied to the target population to obtain the study sample When the process appears sound, one concludes that the results are generalizable

24. 7. View Science as a Cumulative Process Process of Integration  Process of Integration  Systemic Reviews (meta-analysis)  Systemic Reviews (meta-analysis)  Clinical practice guidelines Clinical practice guidelines

25. Popular Statistical Tests T-test - T-test - comparing the means of two independent or matched (related) samples of numerical data ANOVA - three or more independent groups of continuous data differ significantly with regard to a single factor (oneway ANOVA) or two factors (two-way ANOVA) Contingency tables - association between two categorical variables by using the chi-square statistic Survival analysis (Kaplan-Meier) - estimates the probability of an event based on the total period of observation Multivariate (regression) - Multivariate (regression) - Examines the simultaneous effect of multiple predictor variables (generally three or more) on an outcome of interest

26. Statistical Deceptions Standard error is used instead of standard deviation Small sample study results are taken at face value Post hoc P values are used for statistical inference Some results are “significant” but there are a large number of P values Subgroups are compared until statistically significant results are found No significant difference is found between groups in a small sample study Significant P values are crafted through improper use of hypothesis tests

27. The End