From PROMs to PRO-PMs: From instruments to measurements A Role for PROMIS in Performance Improvement Susan Yount, PhD Northwestern University October 26, 2017

What is PROMIS®? Patient-Reported Outcomes Measurement Information System® Measures used to evaluate and monitor physical, mental, and social health (adult & pediatric) Developed and evaluated using state-of-the science psychometric methods Scores on one metric Translations available in Spanish & other languages Now supported by HealthMeasures (healthmeasures.net), the official information & distribution center for 4 NIH-supported measurement systems (PROMIS, Neuro-QoL, NIH Toolbox®, & ASCQ-Me®) Relevant across conditions, domain-specific

Item Response Theory (IRT) Increased in use over past 20+ years for constructing measures of unobservable/latent health constructs (e.g., health status, quality of life) Unique values (parameters) assigned to each item based on how likely people with different levels of the measured trait are to endorse an item IRT-based scores are estimated based on probability model that answers question: Given what is known about the items a person responded to and the pattern of the person’s response, what is the most likely level of the trait (domain) being measured? Advantages: Facilitates evaluation of whether items are equivalent in meaning to different respondents (differential item functioning; DIF) Enables implementing computerized adaptive testing (CAT)

Essential Components of PROMIS DOMAIN The feeling, function, or perception you wish to measure Cuts across different diseases and settings, e.g., physical function, depressive symptoms ITEM BANK Collection of items that each measure the same domain Used to create different measure types, all producing a score on the same metric

Same metric, same meaning Fatigue Item Bank Chemotherapy trial Osteoarthritis trial Heart Failure trial Items 1-10 CAT Items 6-12 Diabetes trial Epilepsy trial Items 2, 4, 9, 13 Items 1-5 Same metric, same meaning

PROMIS Fatigue Across Five Clinical Conditions Cancer w/ benefit (2 mos) Cancer Chemo (B) N = 229 Back Pain (3 mos) Back Pain (1 mo) Back Pain (B) N = 114 Depression (3 mos) Depression (1 mo) Depression (B) N = 64 HF Post-transplant HF Pre-transplant Exacerbation to Stable N = 125 COPD Stable (B) COPD Exacerbation (B) 50 35 40 45 55 60 65 Average for General Population

Assessment at Extremes Person / Item Map Oswestry Disability Index SF36-PF PROMIS PF CAT Excellent PF People Items Excellent PF People Items Excellent PF People Items 6.91% 5.97% 0.81% Ceiling Effects Floor Effects 44.24% 23.65% 3.86% Poor PF Poor PF Poor PF Slide provided by Brodke

PROMIS Measure Types SHORT FORMS COMPUTER ADAPTIVE TESTS (CATs) Subsets of item banks Focused on a single domain Off-the-shelf or custom Usually 4-10 items COMPUTER ADAPTIVE TESTS (CATs) Individually tailored electronic questionnaires Next item administered from item bank depends on previous answers Usually 4-12 items PROFILES Collection of short forms covering multiple domains (e.g. depression, physical function, pain interference) Adult profiles (29, 43, and 57 item versions), Pediatric and Parent-Proxy profiles (25, 37, 49 item versions)

Domains for Adult & Pediatric Assessment

PROMIS is at HealthMeasures PROMIS is at HealthMeasures.net Official information & distribution center for PROMIS, Neuro-QoL, NIH Toolbox, & ASCQ-Me Search & View Measures: Download free, respondent- ready PDFs of PROMIS measures. Hundreds of measures available, all searchable by domain, preferred measure type, language, and more. Applications of HealthMeasures: Free guides to help select PROMIS measures for specific uses or patient populations (e.g., cancer)

Administration & Data Collection Using Paper Respondent-ready PDFs for short forms & profiles on HealthMeasures.net Minimal technology needed; staff time required for data entry and scoring Using a Computer Required for administration of CATs PROMIS measures available in REDCap; Epic 2012, 2016 releases; AOPOC (orthopedics); Assessment CenterSM; OBERD (Outcomes Based Electronic Research Database); Cerner (in dev); Bright Outcome For a custom solution, the Assessment Center API allows your data collection system to administer PROMIS measures (including CATs) Using the PROMIS iPad App Enables on-demand assessments. An administrator can select any PROMIS measure and hand the iPad over to a respondent for completion Available in the iTunes Store

Understanding PROMIS Scores PROMIS measures use a T-score metric 50 is the mean for a relevant reference population 10 is the standard deviation of that population Meaning of the score is defined by how it compares to the scores of others in a reference population High scores equal more of the domain being measured (e.g., more fatigue, more physical function)

PROMIS Score Ranges

PROMIS SCORING OPTIONS HealthMeasures Scoring Service Free, web-based application Upload Excel file w/ raw responses to short form or profile items Receive scored file by email Most accurate and preferred method for calculating scores Automatic Scoring via Data Collection Tool Many data collection tools such as REDCap, Epic, and Assessment Center automatically score responses & deliver results By Hand Scoring Manuals provide details and instructions for scoring individual PROMIS measures Accessed using the Search & View Measures feature and Calculate Scores page on HealthMeasures.net

Linking Measures Converts score from one measure to PROMIS metric Enables comparisons when different measures used www.prosettastone.org Now 24 measures linked

PROMIS in Clinical Practice, Research and Quality Measurement

PROMIS in Clinical Practice University of Rochester 2015: PROMIS CATs administered to nearly every ortho patient, with results viewed immediately in Epic 2017: PROMIS CAT data collected on 95.21% of all ortho patients with >1.1M evaluations collected; expanding to 30 departments/divisions 80% patients complete 3 core domains (PF, PI, Dep; 2.4 min); depts. have option to add up to 5 domains University of Utah Orthopedic surgery: PF, Pain Interference Integrated PROMIS CATs in EHR as part of  mEVAL initiative: patients complete assessment via EHR's patient portal or on tablet at appointment CATs automatically scored, results available to providers within the EHR Partners HealthCare, Boston PROMIS measures in 24 adult and pediatric specialties Goal: expand use across Partners system by end of 2018 (Rotenstein et al., NEJM, 2017: 1.2M PRO scores in 75 clinics across 21 specialties)

PROMIS in Clinical Practice Northwestern: Lurie Distress Screening (Supportive Oncology) Measures completed via MyChart or email link Generates referrals to appropriate providers Cleveland Clinic Neurological Institute University of Washington outpatient HIV Clinic Cincinnati Children’s Hospital Medical Center

PROMIS in Clinical Practice/Research EHR Access to Seamless Integration of PROMIS (EASI-PRO): facilitate collection and use of PRO information to improve clinical care and research (J. Starren, PI) $6.3M NIH/NUCATS grant awarded to coalition of 9 universities, led by NU (U Ill-Chicago, U Chicago, U Alabama-Birmingham, U Kentucky, U Florida , U Utah, Harvard Catalyst CTS, S. California CTSI) Cerner and Epic have signed onto the project as integration collaborators.

PROMIS in Research 2012-2017, PCORI awarded $27 million to studies using PROMIS measures NIH-funded Pediatric Patient Reported Outcomes in Chronic Diseases (PEPR) Consortium: conducting multiple validation studies (asthma, atopic dermatitis, rheumatic disease, cancer, inflammatory bowel disease, sickle cell disease, and diabetes) Registry: Society for Interventional Radiology: as of 4/2017, will use PROMIS for registry for research and quality measurement Clinical trials: as of June 2017, 416 trials in clinicaltrials.gov using PROMIS measures (15% increase from March 2017)

PROMIS in Population Health Over past 10 years, CDC and NIH worked to get NIH PROMIS Global measure added as an HP2020 objective: added in 2015 PROMIS Global in National Health Interview Survey (NHIS), Behavioral Risk Factor Surveillance System (BRFSS), National Health and Nutrition Examination Survey (NHANES) NIH has added PROMIS Global to General Social Survey (GSS) 2018, a nationally representative sample considered appropriate for HP2030 NIH hoping to continue to add NIH PROMIS Global to subsequent administrations of the GSS so trends in HRQOL can be prospectively monitored by HHS As of today, PROMIS Global is the de facto gold standard measure for monitoring HRQOL trends in U.S.

PROMIS as PRO for Performance/Quality Measurement Slide courtesy of National Quality Forum

Guiding Principles: Selection of Patient Reported Outcomes Psychometric Soundness Person-centered Meaningful Actionable Implementable (Cella et al., RTI Press, 2015)

Challenges to Using PROs to Evaluate Care Quality IT challenges to collect, display, interpret PRO data Disruption of clinical workflow PROs not reimbursable Respondent burden Time constraints Linking PROs to clinical interventions Interpretation of PROs Provider unfamiliarity, perceived lack of value Relevance to patients, providers

“Facilitating Patient Reported Outcome Measurement for Key Conditions: Enhancing Universal Measurement for Condition Relevance” (E. Fisher, PI, Dartmouth; PCORI ME-1303-5928) Objective: determine if condition-specific items added to “universal” item banks: Create multi-dimensionality Improve measurement quality Enhance generic measure to be more relevant Adding knee osteoarthritis patient-endorsed condition-specific items to universal item measures: Did not appear to perturb the essential unidimensionality of generic item measures Did not notably improve measure performance F/U cognitive interviews with knee osteoarthritis patients on preferred items and item content will further inform results

PROMIS in Evaluating Care Quality: Environmental Scan (August 2016) Organization Total PROMIS Quality Metric Hit Most Recommended Grant and Contract Funders 36 27 (75%) PROMIS-29, Global-10, Individual Item Banks Professional Societies 17 4 (24%) PROMIS Global-10 Quality Measures Groups 33 17 (52%) Global-10, Individual Item Banks Health Care Systems 34 15 (44%) 120 63 (53%)

PROMIS PF, Pain Int, Dep on all patient visits to foot and ankle clinic between February 2015 and April 2016 PROMIS scores assessed at initial and follow-up visits; minimum clinically important differences (MCIDs) calculated using distribution-based method; receiver operating characteristic (ROC) curves calculated to determine ability of preoperative PROMIS scores to predict MCID Findings: PF scores < 29.7 were likely to improve with surgery, whereas patients with scores > 42 were unlikely to make gains in function Patients with pain scores < 55 were unlikely to improve, whereas those with scores > 67 had clinically significant pain reduction postoperatively Reported prognostic cutoff values help to provide guidance to both the surgeon and the patient and can aid in shared decision making for treatment. Similar assessments for spine surgery, spinal injections, total joint replacements, etc.

University of Utah: another early adopter of PROMIS measures Example: assessed responsiveness of HOOS JR and KOOS JR and PROMIS PF CAT in joint reconstruction 983 patients between 2014-2017, baseline, 3- and/or 6-mo All changes from baseline were significant at 3 and 6 months All measures showed large effect sizes PROMIS PF CAT was consistently most responsive and assessed overall physical functioning (vs. joint-specific) Ass

PROMIS in Evaluating Care Quality The American Academy of Orthopaedic Surgeons recommends PROMIS among Instruments for Collection of Orthopaedic Quality Data The Mayo Clinic recommends and uses PROMIS-10 for Quality of Care Assessment for patients receiving care at Mayo The FDA’s Clinical Outcome Assessment (COA) Compendium includes PROMIS Physical Function in sarcopenia and oncology

PROMIS Pain Interference as Performance/Quality measure PCORI grant (PI: A. Heinemann): Can we demonstrate advantages of using PROMIS as quality metric? PROMIS Pain Interference and Minimum Data Set (MDS) pain items administered to patients in inpatient rehabilitation facility (IRF) 391 patients consented at 2 IRFs; 224 with pain SCI (33%), stroke (26%), TBI (19%) Compared 5/2 PROMIS items to 2 items on MDS 3.0 (interference with ADL, y/n) Findings: 2/5 PROMIS Pain Int items > accurate, reliable than MDS Finer distinctions (% with considerable pain: 30% using MDS, 12% using PROMIS) Implications for decisions on discharging patients with mild, tolerable pain levels

NQF MEASURE INCUBATOR™: Multiple Sclerosis Patient-Reported Outcome Performance Measure Development Industry-funded collaborative effort between NQF, AAN, NU, RTI Purpose: Create an evidence-based PRO-based Performance Measure (PRO-PM) for MS with multi-stakeholder input PRO-PM intended to complement MS outcome PM developed by AAN PRO-PM to become an NQF-endorsed measure appropriate for use in public reporting and reimbursement programs

Challenges of collecting PROs Requires purchase, configuration, storage of devices Displays need to be intuitive, accessible for providers (requires institutional investment in IT) Work needed to better understand scores (normal ranges, thresholds, etc.) Benefits of collecting PROs Improve relationships between physicians and patient Enhance understanding through data Enhance shared decision making through data Enhance workflow efficiency and save time Appropriate selection of PROs (e.g., screening, ROS) allow for more efficient use of time during clinic visit – focus on priorities Facilitate discussion of conversations that otherwise might not take place

The way forward…. How to continue to ensure benefits: Master efficient measurement in clinic Minimize patient burden Display PRO information at point of care Use outcomes predicted from population-level data to inform patient expectations Benefits to patients, providers, systems Surgeons can ID areas where they need improvement, eliminate procedures with less favorable outcomes, avoid performing surgeries on patients unlikely to benefit Enhance patient satisfaction by setting appropriate expectations Placing patient’s voice at forefront of healthcare delivery J. Baumhauer, NEJM, 2017;377:6-9

Thoughts from HealthMeasures User Conference (Sept 27-28, 2017) Enhancing Quality Measurement: Identify meaningful measures: clinicians & patients Finding balance between core and specialty measures critical but challenging (e.g., patient burden) Scores must be available in real-time (in EHR or parallel) Barriers: IT, patient and institutional access, workflow, interpretation PRO-based PM to assess quality of healthcare requires advancement in measurement science PRO scores as measures of quality are being challenged PRO-based quality measures may be appropriate first step (“PRO-process” intermediary?)

Acknowledgments Funding for HealthMeasures was provided by the National Institutes of Health grant U2C CA186878.   PROMIS, Patient-Reported Outcomes Measurement Information System, NIH Toolbox, NIH Toolbox for the Assessment of Neurological and Behavioral Function, ASCQ-Me, Adult Sickle Cell Quality of Life Measurement Information System, and their marks are owned by the U. S. Department of Health and Human Services.

Questions? s-yount@northwestern.edu help@healthmeasures.net