1. From PROMs to PRO-PMs: From instruments to measurements A Role for PROMIS in Performance Improvement Susan Yount, PhD Northwestern University
October 26, 2017

2. What is PROMIS®? Patient-Reported Outcomes Measurement Information System®
Measures used to evaluate and monitor physical, mental, and social health (adult & pediatric)
Developed and evaluated using state-of-the science psychometric methods
Scores on one metric
Translations available in Spanish & other languages
Now supported by HealthMeasures (healthmeasures.net), the official information & distribution center for 4 NIH-supported measurement systems (PROMIS, Neuro-QoL, NIH Toolbox®, & ASCQ-Me®)
Relevant across conditions, domain-specific

3. Item Response Theory (IRT)
Increased in use over past 20+ years for constructing measures of unobservable/latent health constructs (e.g., health status, quality of life)
Unique values (parameters) assigned to each item based on how likely people with different levels of the measured trait are to endorse an item
IRT-based scores are estimated based on probability model that answers question:
Given what is known about the items a person responded to and the pattern of the person’s response, what is the most likely level of the trait (domain) being measured?
Advantages:
Facilitates evaluation of whether items are equivalent in meaning to different respondents (differential item functioning; DIF)
Enables implementing computerized adaptive testing (CAT)

4. Essential Components of PROMIS
DOMAIN The feeling, function, or perception you wish to measure Cuts across different diseases and settings, e.g., physical function, depressive symptoms
ITEM BANK
Collection of items that each measure the same domain
Used to create different measure types, all producing a score on the same metric

5. Same metric, same meaning
Fatigue Item Bank
Chemotherapy trial
Osteoarthritis trial
Heart Failure trial
Items 1-10
CAT
Items 6-12
Diabetes trial
Epilepsy trial
Items 2, 4, 9, 13
Items 1-5
Same metric, same meaning

6. PROMIS Fatigue Across Five Clinical Conditions
Cancer
w/ benefit
(2 mos)
Cancer
Chemo
(B)
N = 229
Back Pain
(3 mos)
Back Pain
(1 mo)
Back Pain
(B)
N = 114
Depression
(3 mos)
Depression
(1 mo)
Depression
(B)
N = 64
HF Post-transplant
HF Pre-transplant
Exacerbation to Stable
N = 125
COPD Stable (B)
COPD Exacerbation (B) 50 35 40 45 55 60 65
Average for General Population

7. Assessment at Extremes Person / Item Map
Oswestry Disability Index
SF36-PF
PROMIS PF CAT
Excellent PF
People Items
Excellent PF
People Items
Excellent PF
People Items
6.91%
5.97%
0.81%
Ceiling Effects
Floor Effects
44.24%
23.65%
3.86%
Poor PF
Poor PF
Poor PF
Slide provided by Brodke

8. PROMIS Measure Types SHORT FORMS COMPUTER ADAPTIVE TESTS (CATs)
Subsets of item banks
Focused on a single domain
Off-the-shelf or custom
Usually 4-10 items
COMPUTER ADAPTIVE TESTS (CATs)
Individually tailored electronic questionnaires
Next item administered from item bank depends on previous answers
Usually 4-12 items
PROFILES
Collection of short forms covering multiple domains (e.g. depression, physical function, pain interference)
Adult profiles (29, 43, and 57 item versions), Pediatric and Parent-Proxy profiles (25, 37, 49 item versions)

9. Domains for Adult & Pediatric Assessment

10. PROMIS is at HealthMeasures
PROMIS is at HealthMeasures.net Official information & distribution center for PROMIS, Neuro-QoL, NIH Toolbox, & ASCQ-Me
Search & View Measures: Download free, respondent- ready PDFs of PROMIS measures. Hundreds of measures available, all searchable by domain, preferred measure type, language, and more.
Applications of HealthMeasures:
Free guides to help select PROMIS measures for specific uses or patient populations (e.g., cancer)

11. Administration & Data Collection
Using Paper
Respondent-ready PDFs for short forms & profiles on HealthMeasures.net
Minimal technology needed; staff time required for data entry and scoring
Using a Computer
Required for administration of CATs
PROMIS measures available in REDCap; Epic 2012, 2016 releases; AOPOC (orthopedics); Assessment CenterSM; OBERD (Outcomes Based Electronic Research Database); Cerner (in dev); Bright Outcome
For a custom solution, the Assessment Center API allows your data collection system to administer PROMIS measures (including CATs)
Using the PROMIS iPad App
Enables on-demand assessments. An administrator can select any PROMIS measure and hand the iPad over to a respondent for completion
Available in the iTunes Store

12. Understanding PROMIS Scores
PROMIS measures use a T-score metric
50 is the mean for a relevant reference population
10 is the standard deviation of that population
Meaning of the score is defined by how it compares to the scores of others in a reference population
High scores equal more of the domain being measured (e.g., more fatigue, more physical function)

13. PROMIS Score Ranges

14. PROMIS SCORING OPTIONS
HealthMeasures Scoring Service
Free, web-based application
Upload Excel file w/ raw responses to short form or profile items
Receive scored file by
Most accurate and preferred method for calculating scores
Automatic Scoring via Data Collection Tool
Many data collection tools such as REDCap, Epic, and Assessment Center automatically score responses & deliver results
By Hand
Scoring Manuals provide details and instructions for scoring individual PROMIS measures
Accessed using the Search & View Measures feature and Calculate Scores page on HealthMeasures.net

15. Linking Measures Converts score from one measure to PROMIS metric
Enables comparisons when different measures used
Now 24 measures linked

16. PROMIS in Clinical Practice, Research and Quality Measurement

17. PROMIS in Clinical Practice
University of Rochester
2015: PROMIS CATs administered to nearly every ortho patient, with results viewed immediately in Epic
2017: PROMIS CAT data collected on 95.21% of all ortho patients with >1.1M evaluations collected; expanding to 30 departments/divisions
80% patients complete 3 core domains (PF, PI, Dep; 2.4 min); depts. have option to add up to 5 domains
University of Utah
Orthopedic surgery: PF, Pain Interference
Integrated PROMIS CATs in EHR as part of  mEVAL initiative: patients complete assessment via EHR's patient portal or on tablet at appointment
CATs automatically scored, results available to providers within the EHR
Partners HealthCare, Boston
PROMIS measures in 24 adult and pediatric specialties
Goal: expand use across Partners system by end of 2018
(Rotenstein et al., NEJM, 2017: 1.2M PRO scores in 75 clinics across 21 specialties)

18. PROMIS in Clinical Practice
Northwestern: Lurie Distress Screening (Supportive Oncology)
Measures completed via MyChart or link
Generates referrals to appropriate providers
Cleveland Clinic Neurological Institute
University of Washington outpatient HIV Clinic
Cincinnati Children’s Hospital Medical Center

19. PROMIS in Clinical Practice/Research
EHR Access to Seamless Integration of PROMIS (EASI-PRO): facilitate collection and use of PRO information to improve clinical care and research (J. Starren, PI)
$6.3M NIH/NUCATS grant awarded to coalition of 9 universities, led by NU (U Ill-Chicago, U Chicago, U Alabama-Birmingham, U Kentucky, U Florida , U Utah, Harvard Catalyst CTS, S. California CTSI)
Cerner and Epic have signed onto the project as integration collaborators.

20. PROMIS in Research
, PCORI awarded $27 million to studies using PROMIS measures
NIH-funded Pediatric Patient Reported Outcomes in Chronic Diseases (PEPR) Consortium: conducting multiple validation studies (asthma, atopic dermatitis, rheumatic disease, cancer, inflammatory bowel disease, sickle cell disease, and diabetes)
Registry: Society for Interventional Radiology: as of 4/2017, will use PROMIS for registry for research and quality measurement
Clinical trials: as of June 2017, 416 trials in clinicaltrials.gov using PROMIS measures (15% increase from March 2017)

21. PROMIS in Population Health
Over past 10 years, CDC and NIH worked to get NIH PROMIS Global measure added as an HP2020 objective: added in 2015
PROMIS Global in National Health Interview Survey (NHIS), Behavioral Risk Factor Surveillance System (BRFSS), National Health and Nutrition Examination Survey (NHANES)
NIH has added PROMIS Global to General Social Survey (GSS) 2018, a nationally representative sample considered appropriate for HP2030
NIH hoping to continue to add NIH PROMIS Global to subsequent administrations of the GSS so trends in HRQOL can be prospectively monitored by HHS
As of today, PROMIS Global is the de facto gold standard measure for monitoring HRQOL trends in U.S.

22. PROMIS as PRO for Performance/Quality Measurement
Slide courtesy of National Quality Forum

23. Guiding Principles: Selection of Patient Reported Outcomes
Psychometric Soundness
Person-centered
Meaningful
Actionable
Implementable
(Cella et al., RTI Press, 2015)

24. Challenges to Using PROs to Evaluate Care Quality
IT challenges to collect, display, interpret PRO data
Disruption of clinical workflow
PROs not reimbursable
Respondent burden
Time constraints
Linking PROs to clinical interventions
Interpretation of PROs
Provider unfamiliarity, perceived lack of value
Relevance to patients, providers

25. “Facilitating Patient Reported Outcome Measurement for Key Conditions: Enhancing Universal Measurement for Condition Relevance” (E. Fisher, PI, Dartmouth; PCORI ME )
Objective: determine if condition-specific items added to “universal” item banks:
Create multi-dimensionality
Improve measurement quality
Enhance generic measure to be more relevant
Adding knee osteoarthritis patient-endorsed condition-specific items to universal item measures:
Did not appear to perturb the essential unidimensionality of generic item measures
Did not notably improve measure performance
F/U cognitive interviews with knee osteoarthritis patients on preferred items and item content will further inform results

26. PROMIS in Evaluating Care Quality: Environmental Scan (August 2016)
Organization
Total
PROMIS Quality Metric Hit
Most Recommended
Grant and Contract Funders 36
27 (75%)
PROMIS-29, Global-10, Individual Item Banks
Professional Societies 17
4 (24%)
PROMIS Global-10
Quality Measures Groups 33
17 (52%)
Global-10, Individual Item Banks
Health Care Systems 34
15 (44%)
120
63 (53%)

27. PROMIS PF, Pain Int, Dep on all patient visits to foot and ankle clinic between February 2015 and April 2016
PROMIS scores assessed at initial and follow-up visits; minimum clinically important differences (MCIDs) calculated using distribution-based method; receiver operating characteristic (ROC) curves calculated to determine ability of preoperative PROMIS scores to predict MCID
Findings:
PF scores < 29.7 were likely to improve with surgery, whereas patients with scores > 42 were unlikely to make gains in function
Patients with pain scores < 55 were unlikely to improve, whereas those with scores > 67 had clinically significant pain reduction postoperatively
Reported prognostic cutoff values help to provide guidance to both the surgeon and the patient and can aid in shared decision making for treatment.
Similar assessments for spine surgery, spinal injections, total joint replacements, etc.

28. University of Utah: another early adopter of PROMIS measures
Example: assessed responsiveness of HOOS JR and KOOS JR and PROMIS PF CAT in joint reconstruction
983 patients between , baseline, 3- and/or 6-mo
All changes from baseline were significant at 3 and 6 months
All measures showed large effect sizes
PROMIS PF CAT was consistently most responsive and assessed overall physical functioning (vs. joint-specific)
Ass

29. PROMIS in Evaluating Care Quality
The American Academy of Orthopaedic Surgeons recommends PROMIS among Instruments for Collection of Orthopaedic Quality Data
The Mayo Clinic recommends and uses PROMIS-10 for Quality of Care Assessment for patients receiving care at Mayo
The FDA’s Clinical Outcome Assessment (COA) Compendium includes PROMIS Physical Function in sarcopenia and oncology

30. PROMIS Pain Interference as Performance/Quality measure
PCORI grant (PI: A. Heinemann): Can we demonstrate advantages of using PROMIS as quality metric?
PROMIS Pain Interference and Minimum Data Set (MDS) pain items administered to patients in inpatient rehabilitation facility (IRF)
391 patients consented at 2 IRFs; 224 with pain
SCI (33%), stroke (26%), TBI (19%)
Compared 5/2 PROMIS items to 2 items on MDS 3.0 (interference with ADL, y/n)
Findings:
2/5 PROMIS Pain Int items > accurate, reliable than MDS
Finer distinctions (% with considerable pain: 30% using MDS, 12% using PROMIS)
Implications for decisions on discharging patients with mild, tolerable pain levels

31. NQF MEASURE INCUBATOR™: Multiple Sclerosis Patient-Reported Outcome Performance Measure Development
Industry-funded collaborative effort between NQF, AAN, NU, RTI
Purpose:
Create an evidence-based PRO-based Performance Measure (PRO-PM) for MS with multi-stakeholder input
PRO-PM intended to complement MS outcome PM developed by AAN
PRO-PM to become an NQF-endorsed measure appropriate for use in public reporting and reimbursement programs

32. Challenges of collecting PROs
Requires purchase, configuration, storage of devices
Displays need to be intuitive, accessible for providers (requires institutional investment in IT)
Work needed to better understand scores (normal ranges, thresholds, etc.)
Benefits of collecting PROs
Improve relationships between physicians and patient
Enhance understanding through data
Enhance shared decision making through data
Enhance workflow efficiency and save time
Appropriate selection of PROs (e.g., screening, ROS) allow for more efficient use of time during clinic visit – focus on priorities
Facilitate discussion of conversations that otherwise might not take place

33. The way forward…. How to continue to ensure benefits:
Master efficient measurement in clinic
Minimize patient burden
Display PRO information at point of care
Use outcomes predicted from population-level data to inform patient expectations
Benefits to patients, providers, systems
Surgeons can ID areas where they need improvement, eliminate procedures with less favorable outcomes, avoid performing surgeries on patients unlikely to benefit
Enhance patient satisfaction by setting appropriate expectations
Placing patient’s voice at forefront of healthcare delivery
J. Baumhauer, NEJM, 2017;377:6-9

34. Thoughts from HealthMeasures User Conference (Sept 27-28, 2017)
Enhancing Quality Measurement:
Identify meaningful measures: clinicians & patients
Finding balance between core and specialty measures critical but challenging (e.g., patient burden)
Scores must be available in real-time (in EHR or parallel)
Barriers: IT, patient and institutional access, workflow, interpretation
PRO-based PM to assess quality of healthcare requires advancement in measurement science
PRO scores as measures of quality are being challenged
PRO-based quality measures may be appropriate first step (“PRO-process” intermediary?)

35. Acknowledgments
Funding for HealthMeasures was provided by the National Institutes of Health grant U2C CA
PROMIS, Patient-Reported Outcomes Measurement Information System, NIH Toolbox, NIH Toolbox for the Assessment of Neurological and Behavioral Function, ASCQ-Me, Adult Sickle Cell Quality of Life Measurement Information System, and their marks are owned by the U. S. Department of Health and Human Services.

36. Questions? s-yount@northwestern.edu help@healthmeasures.net