David C. Chang, PhD, MPH, MBA Director of Outcomes Research UCSD Department of Surgery Introduction to Outcomes Research Methods and Data Resources

Surgery and public health

Problem in surgical clinical research Unregulated FDA regulation applies only to “devices” (whether a real device, or a molecular device in the form of a drug) Procedural medicine are not regulated Many reasons: complexity, difficulty in standardizing, difficulty of enforcement (“surgeons know best” attitude) Self-regulation

Erroneous literature

RCTs often too late “Tipping Point” EVAR-1, DREAM OVER

Social responsibility It is our responsibility in academic medicine, to shoulder the responsibility that, in other fields of medicine, has been assumed by the FDA To ensure that only good treatment modalities are applied to patients

Biggest barrier to good research? Not having a correctly constructed hypothesis Incorrect design Don’t know how to get data Fear of statistics

Typical questions Components What/why/when/how Verb Condition “Why is the sky blue?” “What is the typical presentation of appendicitis?” Open-ended

Open-ended questions Descriptive analysis Observational study = no comparison = no statistical test Only one denominator May have more than one numerator, generating more than one ratio All ratios are calculated with the same denominator

Descriptive statistics P value not applicable to compare different parts of the same population

Value and pitfall To explore the unknown When you know nothing, the first step is to explore and document the numbers Risk of over-generalizing

Inferential statistics P value applicable for comparing parts of two populations

What is a hypothesis? Question ≠ hypothesis Questions: usually open-ended Hypothesis: usually is closed-ended, asking for a yes/no answer Statistical testing can only give yes/no answers

The process – study design Study design phaseData preparationAnalysis phase Question developmentSelect databaseUnivariate Define populationLink databaseBivariate Define subsetSelect data elementsMultivariable Define outcomeGenerate new data elementsSensitivity Define primary comparisonSubset analysis Define covariates

Steps in constructing a hypothesis Specify the outcomes (O in PICO) Common oversight: Often focus on the P, but vague about O (a typical question, “What is the outcome (?) of xyz patients?”) Specify the comparisons (C in PICO) Not done in open-ended questions Specify covariates (control variables, adjustment)

Hypothesis statement y = b1X1 + b2X2 + b3X3 Death = age + race + gender + insurance…

Inclusion/exclusion criteria Just like a clinical trials (“eligibility criteria”) Diagnosis and/or procedure codes? Common mistake

Comparison

Outcome Mortality? Rare Complications Length of stay Charges Be judicious

Covariates / independent variables Patient demographcis Patient comorbidity Surgeon volume Hospital volume Hospital type (teaching vs non-teaching) Area (rural vs urban)

Hierarchy of influence on surgical outcomes Technique and Management Patient Surgeon Hospital Region Nation Outcomes research Clinical trials

The process – data preparation Study design phaseData preparationAnalysis phase Question developmentSelect databaseUnivariate Define populationLink databaseBivariate Define subsetSelect data elementsMultivariable Define outcomeGenerate new data elementsSensitivity Define primary comparisonSubset analysis Define covariates

Overview of public and semi-public databases Multi-specialty Administrative Databases Nationwide Inpatient Sample (NIS) Medicare, Medicaid California OSHPD Clinical Databases National Surgical Quality Improvement Program (NSQIP) Specialty-specific Trauma National Trauma Databank (NTDB) Oncology Surveillance, Epidemiology, and End Results (SEER) National Cancer Databank (NCDB) Transplant United Network for Organ Sharing (UNOS)

Administrative databases Advantages Large patient numbers Less selection bias Can be linked to other databases containing other non- medical information Disadvantages Limited clinical course information Limited surgical procedure information

NSQIP/non-NSQIP in-hospital mortality

Select data elements

Generate new data elements Most time consuming step of outcomes analysis Not every component of your research question is readily available in the database For example, comorbidity Charlson Index, Elixhauser Index Some common concepts actually undefined Readmission?

What is a “re-admission”? Not all “admissions” are “re-admissions” 30-day? Elective? Transfers? Diagnosis-specific? Preventable?

The process – analysis Study design phaseData preparationAnalysis phase Question developmentSelect databaseUnivariate Define populationLink databaseBivariate Define subsetSelect data elementsMultivariable Define outcomeGenerate new data elementsSensitivity Define primary comparisonSubset analysis Define covariates

Hypothesis statement y = b1X1 + b2X2 + b3X3 Death = age + race + gender + insurance…

Table 1: Descriptive analysis

Table 2: Bi-variate analysis (unadjusted comparison)

Table 3: Multivariable analysis (adjusted analysis)

Analysis for Table 1

P value not applicable to compare different parts of the same population

Analysis for Table 1 % for categorical data Mean/median/SD for continuous data For exploratory studies, descriptive studies, case series, etc., this would be the end of the process Reminder, avoid overgeneralizing

Analysis for Table 2

Think about data types… Continuous data Categorical data (Ordinal data)

Analysis for Table 2 Two questions to think about when picking a stats test… What is my outcome/dependent variable? What is my independent/input variable? What type of data do I have for each? 4 possible combinations: 2 variables 2 data types

X = input Y = outcome Cat. Cont. T-test Rank sum ROC 22 Correlation Analysis for Table 2

Analysis for Table 3

X = input Y = outcome Cat. Cont. Logistic regression Linear regression T-test Rank sum ROC 22 Correlation Analysis for table 3

Subset analysis Consistency of findings Generalizability

“This is not research anymore”

“That guy”