Composite Score Methods to Evaluate Part D Organization's Performance over multiple Measures For American Public Health Association (APHA) Annual Conference Philadelphia, Nov 10, 2009 Presenters: Ying Wang, PhD, IMPAQ International (IMPAQ); Sungsoo Oh, MS, Centers for Medicare & Medicaid Services (CMS) Co-authors: Arthur Kirsch, PhD, IMPAQ; Christopher Powers, PharmD, CMS; Oswaldo Urdapilleta, PhD, IMPAQ; and Vikki Oates, CMS SPONSOR: Centers for Medicare & Medicaid Services (CMS)

Presenter Disclosure Presenters: Ying Wang & Sungsoo Oh The following personal financial relationships with commercial interests relevant to this presentation existed during the past 12 months: No relationships to disclose 2

Background Medicare Modernization Act (MMA) and Medicare Part D program (Since 2003) – Medicare Advantage plans that offer prescription drug benefits (MA-PD) – Stand-alone prescription drug plans (PDP) CMS roadmap in quality improvement – Value-based purchasing (VBP) – Quality and performance measurement – Transparency and consumer choice empowerment 3

Project Objective Report card system: Use star ratings to evaluate Part D organization’s performance – along with the rating system for other health care providers (e.g. hospitals, nursing homes, etc.) – Individual measure star – Domain level star: Group related measures into domains – Summary score: Summarize all measure stars into a single rating Use grouping methodology to collapse individual performance measures into domains, and use domain and summary score to rank health plans, and to reduce the level of information beneficiaries need to absorb 4

Data CMS Part D performance measures for 2009 open enrollment which started in Nov, 2008 – therefore, this analysis is using 2008’s data. 545 Part D contracts (462 MA-PD; 83 PDP) Primary data sources for measures: – Call center operation data – Complaints Tracking Module (CTM) – Consumer Assessment of Healthcare Providers and Systems (CAHPS) – Data forwarded by Independent Review Entity (IRE) – Management Information Integrated Repository (MIIR) – Medicare Beneficiary Database (MBD) – Medicare Advantage Prescription Drug System (MARx) 5

Data – Performance Measures Based on CMS’s grouping structure used for 2009’s open enrollment (Started from Nov, 2008): Drug Plan Customer Service  Time on Hold When Customer Calls Drug Plan  Calls Disconnected When Customer Calls Drug Plan  Time on Hold When Pharmacist Calls Drug Plan  Calls Disconnected When Pharmacist Calls Drug Plan  Complaints about the Drug Plan  How Helpful Is Your Plan When You Need Information  Rating of Drug Plan Using Your Plan to Get Your Prescriptions Filled  Rating of Drug Plan  Pharmacists Have Up-to-date Information on Plan Members Who Need Extra Help  Complaints about the Plan’s Benefits and Access to Prescription Drugs  Complaints about Joining and Leaving the Plan  Delays in Appeals Decisions  Reviewing Appeals Decisions Drug Pricing Information  Availability of Drug Coverage and Cost Information  How Often the Plan’s Drug Prices Change  Complaints about the Plan’s Drug Pricing and Out-of-pocket Costs 6

Grouping Methods Literature review of the prospective grouping methods – exploratory factor analysis (EFA) – confirmatory factor analysis (CFA) – principal component analysis (PCA) – cluster analysis – common/principal factor analysis (PFA) Strength and Limitations of each method Identified PFA and cluster analysis passed initial screening and were chosen for actual data analyses 7

Analysis Guidelines from CMS Missing data should not be imputed because of the lack of correlation among contracts, but approaches to deal with missing data should be explored. The same grouping structure should be applied to both MA-PD and PDP. Unsuppressed data and raw scores should be utilized to test the groupings. Since several variables are rates based on a denominator of enrollment with an exclusion rule of 800, some analyses is based only on contracts with 800 or more enrollees. For measures that are not loaded into factors, CMS reserves the right to assign them to groups subjectively. Three groups would be preferred as it is consistent with the previous year’s domain number. 8

Data Problems and Remedies Problems – Large missing data with CAHPS (50%~) – Measures with skewed distribution – Correlation between some measures are too high Remedies – Treated CAHPS measures separately – Log-transformed some measures that are skewed – Dropped one measure if highly correlated w/ the other – Sensitivity analysis, i.e.- tested the different choices of analytical datasets and analysis methods 9

Other Decision Points in Analysis Principal Factor Analysis (PFA) – Selection of the factor model – Selection of the factor rotation technique – Choice of the factor significance level – Determination of the number of factors to retain Cluster Analysis – Choice of clustering methods – Determination of the optimal number of clusters PFA as the primary grouping method, and Cluster Analysis as the validation tool 10

Findings – PFA results 11

Findings – Cluster Analysis Cluster analyses were performed on the similar datasets used for PFA. Similar results (grouping structure) were observed by cluster analysis, which verified the PFA results. 12

Conclusions Although some measures seem to be difference between MA- PD and PDP contracts, based on simple t tests among two groups, the analyses by either the combined data or MA-PD alone generated similar results in both the principal factor analysis and the cluster analysis. Among the three groups derived, one group (factor 3) is not stable and the internal consistency reliability is relatively low. Dropping the fuzz value and relaxing the number of groups parameter does not improve the factor loadings. The small number of contracts did challenge the stability and significance level of factor loadings. 13

Conclusions (Continued) The large amount of missing data associated with the CAHPS measures creates a real analytical problem if they are combined with the other measures in analysis. Several measures were not loaded into any of the three primary factors, even after several adjustments were made to the model. Although these measures ended up loading into different groups in various sensitivity analyses, the internal consistency reliability was significantly jeopardized. In setting the number of groups to 3, both the factor analysis and cluster analysis generated very similar results as well. 14

Conclusions (Continued) CAHPS measures should be reported separately from the groupings of the other variables. For measures not loaded into any groups, CMS is interested in subjectively assigning them to certain group/domain by priori. Naming of the factors is largely driven by the measure(s) that have higher loadings in their respective groups. The statistical analysis does not load all measures into groups, but to make the best use of all performance measures, a policy decision will be necessary to assign measures appropriately to the final grouping structure, based on subject matter expertise. 15

Conclusions (Continued) 16 Some new performance measures were later added and older measures retired during the method development process. The final grouping structure chosen for 2009 open enrollment and public reporting is below:

Next Step Use more recent performance data to re-run the analysis and further test the grouping structure Merge in more data at different time points to create a longitudinal data to test the robustness of the data-driven grouping structure 17

Thank you! Additional questions/comments, see us afterwards … 18