1. Standardizing VBP's Patient Satisfaction Data into Percentile Ranks Using Reference Distribution Jenhao Jacob Cheng, PhD, MS Alice Li, MS Nikolas Matthes, MD, PhD, MPH Quality Indicator Project Press Ganey Associates, Inc. jcheng@pressganey.com APHA Annual Meeting, Denver CO, November 10 2010

2. 2 Background –The 2010 Heath Care Reform Bill mandates the implementation of Medicare’s value-based Purchasing (VBP) program for fiscal year 2013 –VBP evaluates hospitals’ performance by 17 clinical measures for process of care and 9 patient satisfaction measures on a rate-based (percent) scale –Prior year’s performance serves as the baseline reference from which the benchmarks (median and 95 th percentile) are derived for assessment year period –Patient satisfaction data must be converted into percentile rank (0-100) based on baseline data to facilitate the scoring process

3. Percentile Rank –People usually get confused between percentile rate (in original data scale) and percentile ranks (always 0-100) –Typically we are interested to know a data distribution by percentile rates on some commonly used percentile ranks (e.g., 25, 50, 75) –Sometimes we convert the data into percentile ranks, like in this study, to reflect the position in the reference distribution for comparison purpose –Therefore, percentile rank is a standardized score to tell how good you are for any of the target measures –Determining percentile ranks within one dataset is easy but is more challenging when two datasets involved 3

4. 4 Scoring for Patient Satisfaction Patient Satisfaction MeasureRatePercentile RankScore 1. Nurse Communication0 - 10 - 1000 - 10 2. Doctor Communication0 - 10 - 1000 - 10 3. Cleanliness and Quiet0 - 10 - 1000 - 10 4. Staff Responsiveness0 - 10 - 1000 - 10 5. Medications Communication0 - 10 - 1000 - 10 6. Pain Management0 - 10 - 1000 - 10 7. Discharge Information0 - 10 - 1000 - 10 8. Overall Hospital Rating0 - 10 - 1000 - 10 9. Lowest Percentile Rank 0 - 1000 - 20 Total0 - 100 First 8 measures are assigned a score up to 10 points based on percentile ranks and up to 20 additional points are awarded for the lowest percentile rank

5. 5 Study Design –Retrospective cohort study based on hospital- level data. –Study population: 3,725 hospitals with HCAHPS data for both 2007 and 2008 from Hospital Compare website, maintained by Centers for Medicare and Medicaid Services (CMS) –For demonstration purposes only one patient satisfaction measure, Overall Hospital Rating, has been used.

6. 6 Methodology –We investigated three methods to assign percentile ranks based on reference distribution: Cross Ranking (CR) Percentile Mapping (PM) Distribution Modeling (DM) –Each method is individually evaluated for accuracy, variation, and efficiency –Pairwise comparison is performed for each pair of methods to evaluate agreement with each other

7. Cross Ranking 7 –Ranking each data point repeatedly within the entire reference distribution, implemented by SAS Proc SQL –When using properly, SQL can be viewed as a rank-based statistical tool for data analysis –The feature of Cartesian product allows SQL to handle the repeated ordering of each data point (Y i ) in reference distribution (X) –Percentile rank of each data point is calculated by Count (X<Y i ) / N –N x N processing

8. SQL Code: Cartesian Product select y.rate, sum(case when x.rate<=y.rate then 1 else 0 end) as num, count(*) as den, 100*(calculated num / calculated den) as pctl from x, y group by id, y.rate; 8

9. Percentile Mapping –Create a percentile lookup table which contains 101 pairs of “rank” and “rate” listings, implemented by SAS Proc UNIVARIATE (or Oracle percentile function) –Mapping each data point (rate) to the nearest percentile value (rate) to obtain the corresponding percentile rank –Mapping can be implemented by a compound SQL technique called “Nearest Neighbor Join” which involves two “Unequal Join” procedures in opposite directions –Two unequal joins not only make sure the nearest mapping is obtained but also take care of the potential outlier in either direction of unequal join –N x 101 processing 9

10. SQL Code: Nearest Neighbor Join select rate, pctl from ( select y.rate, max(look.pctl) as pctl, min(abs(y.rate-look.rate)) as diff from y, look where y.rate>=look.rate group by id, y.rate union select y.rate, min(look.pctl) as pctl, min(abs(y.rate-look.rate)) as diff from y, look where y.rate<=look.rate group by id, y.rate ) having diff=min(diff); 10

11. Distribution Modeling –Compute the empirical cumulative distribution function (CDF) based on Kernel density estimation, implemented by SAS Proc KDE –Fitting CDF data by piecewise linear regression with 9 knots, implemented by SAS Proc REG –Note that the rounded predictions represent percentile ranks –Increase # of knots or include quadratic functions can improve the model fit but the gain is limited –Cubic functions are not suggested to avoid fitting a decreasing curve section 11

12. 12 Modeling CDF Curve

13. 13 Comparison Analysis –Accuracy of each method is evaluated by Kendall’s t coefficient for rank correlation between percentile rank and original data (rate) –Variation of each method is evaluated by the standard deviation –Efficiency of each method is evaluated by computation time using SAS 9.2 on a reference desktop computer –Pairwise agreement is evaluated by root mean square error (RMSE) between each pair of methods

14. Individual ComparisonPairwise Comparison MethodKendall tSDTime [sec]MethodRMSE CR0.999628.7134.7CR-PM0.68 PM0.999728.670.25CR-DM0.89 DM0.999828.220.13PM-DM0.79 14 Comparison Results

15. 15 Conclusion –DM is the most parsimonious and efficient method to convert data into percentile ranks based on reference distribution –Its statistical properties in terms of accuracy and variation are also a bit superior but requires more mathematical knowledge to understand the approach –CR can be easily implemented by standard IT solution like SQL but is much more computationally intensive and this may prevent it to be used as an online tool –PM presents an in-between choice which is still efficient for online applications but not that difficult for non- statistician (IT) to understand and implement it

16. 16 Discussion –Statistical properties like accuracy and variation are not very different since they simply reflect how smoothly the percentile ranks are calculated –Therefore, they can be more different when N is smaller and/or many ties exist –On the other hand, efficiency is more distinguishing in this study when N is large and vise versa –PM is conceptually similar to DM but its calculation is more like CR in a discrete domain (while DM in continuous domain) –Pairwise comparison also confirms that CR and PM are in more agreement