Onsite Quarterly Meeting SIPP PIPs June 13, 2012 Presenter: Christy Hormann, LMSW, CPHQ Project Leader-PIP Team

Overview of Presentation  Progression of SIPP PIPs  SFY 2012 validation results  Areas for improvement

SFY 2011 SIPP PIPs  First year SIPPs completed the PIP process  Each SIPP required to submit one PIP  Total of 14 PIPs were submitted

SFY 2011 SIPP PIPs cont.  One was completed through Activity VI  13 were completed through VIII  Initial scores were lower due to lack of the proper documentation

SFY 2012 SIPP PIPs  For SFY 2012, the SIPPs were required to submit the collaborative PIP and a PIP topic of their choosing  Some of the individual SIPP topics were increasing family participation in treatment, minimizing weight gain during treatment, and reducing readmissions

SFY 2012 SIPP PIPs cont.  There were a total of 27 PIPs submitted for validation  Six PIPs were assessed through Activity VI  Four PIPs were assessed through Activity VII  Five PIPs were assessed through Activity VIII

SFY 2012 SIPP PIPs cont.  Twelve PIPs were assessed through Activity IX  None of the PIPs were assessed for Activity X-Sustained Improvement

PIP Stages

Study Design Stage  Establishes methodological framework for the PIP  Includes development of study topic, question, indicators, and population (Activities I through IV)  A strong study design is necessary for the successful progression of a PIP

Study Design Stage Evaluation Elements Activity I: Study Topic  Reflects high-volume or high-risk conditions  Is selected following collection and analysis of data

Study Design Stage Evaluation Elements Activity I: Study Topic  Addresses a broad spectrum of care and services  Includes all eligible populations that meet the study criteria  Does not exclude members with special health care needs  Has the potential to affect member health, functional status, or satisfaction

Study Design Stage Evaluation Elements Activity II: Study Question  States the problem to be studied in simple terms  Is answerable

Study Design Stage Evaluation Elements Activity III: Study Indicators  Are well-defined, objective, and measurable  Are based on current, evidence-based practice guidelines, pertinent peer-reviewed literature, or consensus expert panels  Allow for the study question to be answered

Study Design Stage Evaluation Elements Activity III: Study Indicators  Measure changes (outcomes) in health or functional status, member satisfaction, or valid process alternatives  Have available data that can be collected on each indicator

Study Design Stage Evaluation Elements Activity III: Study Indicators  Are nationally recognized measures, such as HEDIS technical specifications, when appropriate  Includes the basis on which indicator(s) was adopted, if internally developed

Study Design Stage Evaluation Elements Activity IV: Study Population  Is accurately and completely defined  Includes requirements for the length of a member’s enrollment in the MCO  Captures all members to whom the study question applies

SIPP Design Stage Results Study StageActivityMet Partially Met Not Met Design I. Appropriate Study Topic* 87% (139/160) 3% (4/160) 11% (17/160) II. Clearly Defined, Answerable Study Question(s) 74% (40/54) 11% (6/54) 15% (8/54) III. Clearly Defined Study Indicator(s) * 68% (86/127) 9% (11/127) 24% (30/127) IV. Correctly Identified Study Population 75% (48/64) 14% (9/64) 11% (7/64) Design Total * 77% (313/405) 7% (30/405) 15% (62/405) * The activity or stage total may not equal 100 percent due to rounding.

Study Implementation Stage  Includes sampling, data collection, and interventions (Activities V through VII)  During this stage, MCOs collect data, evaluate and identify barriers to performance, and development interventions targeted to improve outcomes  The implementation of effective improvement strategies is necessary to improve PIP outcomes

Study Implementation Stage Evaluation Elements Activity V: Sampling  Consider and specify the true or estimated frequency of occurrence  Identify the sample size  Specify the confidence level  Specify the acceptable margin of error

Study Implementation Stage Evaluation Elements Activity V: Sampling  Ensure a representative sample of the eligible population  Are in accordance with generally accepted principles of research design and statistical analysis

Study Implementation Stage Evaluation Elements Activity VI: Data Collection  The identification of data elements to be collected  The identification of specified sources of data  A defined and systematic process for collecting baseline and remeasurement data  A timeline for the collection of baseline and remeasurement data

Study Implementation Stage Evaluation Elements Activity VI: Data Collection  Qualified staff and personnel to abstract manual data  A manual data collection tool that ensures consistent and accurate collection of data according to indicator specifications  A manual data collection tool that supports interrater reliability

Study Implementation Stage Evaluation Elements Activity VI: Data Collection  Clear and concise written instructions for completing the manual data collection tool  An overview of the study in written instructions

Study Implementation Stage Evaluation Elements Activity VI: Data Collection  Administrative data collection algorithms/ flow charts that show activities in the production of indicators  An estimated degree of administrative data completeness

Study Implementation Stage Evaluation Elements Activity VII: Interventions  Related to causes/barriers identified through data analysis and quality improvement processes  System changes that are likely to induce permanent change  Revised if the original interventions are not successful  Standardized and monitored if interventions are successful

SIPP Implementation Stage Results Study StageActivity Met Partially MetNot Met Implementation V. Valid Sampling Techniques (if sampling was used) 86% (6/7) 0% (0/7) 14% (1/7) VI. Accurate/Complete Data Collection * 58% (159/272) 13% (36/272) 28% (77/272) VII. Appropriate Improvement Strategies 70% (45/64) 11% (7/64) 19% (12/64) Implementation Total 61% (210/343) 13% (43/343) 26% (90/343) * The activity or stage total may not equal 100 percent due to rounding.

Outcomes Stage  The final stage of the PIP process (Activities VIII through X)  Involves data analysis and the evaluation of improvement based on the reported results and statistical testing  Sustained improvement is achieved when outcomes exhibit improvement over multiple measurements

Outcomes Stage Evaluation Elements Activity VIII: Data Analysis  Are conducted according to the data analysis plan in the study design  Allow for the generalization of results to the study population if a sample was selected  Identify factors that threaten the internal or external validity of findings  Include an interpretation of findings

Outcomes Stage Evaluation Elements Activity VIII: Data Analysis  Are presented in a way that provides accurate, clear, and easily understood information  Identify the initial measurement and the remeasurement of the study indicators  Identify statistical differences between the initial measurement and the remeasurement

Outcomes Stage Evaluation Elements Activity VIII: Data Analysis  Identify factors that affect the ability to compare the initial measurement with the remeasurement  Include an interpretation of the extent to which the study was successful

Outcomes Stage Evaluation Elements Activity IX: Real Improvement  The remeasurement methodology is the same as the baseline methodology  There is documented improvement in processes or outcomes of care  The improvement appears to be the result of planned intervention(s)  There is statistical evidence that observed improvement is true improvement

Outcomes Stage Evaluation Elements Activity X: Sustained Improvement  Repeated measurements over comparable time periods demonstrate sustained improvement or that a decline in improvement is not statistically significant

SIPPs Outcomes Stage Results Study StageActivity Met Partially MetNot Met Outcomes VIII. Sufficient Data Analysis and Interpretation * 54% (63/116) 17% (20/116) 28% (33/116) IX. Real Improvement Achieved 54% (26/48) 21% (10/48) 25% (12/48) X. Sustained Improvement Achieved‡‡‡ Outcomes Total * 54% (89/164) 18% (30/164) 27% (45/164) * The activity or stage total may not equal 100 percent due to rounding. ‡ The PIPs did not progress to this phase during the review period and could not be assessed for real or sustained improvement.

SIPP Indicator Results  There were a total of 44 study indicators  22 were not assessed for improvement  15 demonstrated improvement  Of those that demonstrated improvement, 11 demonstrated statistically significant improvement

SIPP Indicator Results SFY 2012 Performance Improvement Project Outcomes for the SIPPs (N=27 PIPs) SIPPs Total Number of Study Indicators Comparison to Study Indicator Results from Prior Measurement Period Sustained Improvement 1 Declined Statistically Significant Decline Improved Statistically Significant Improvement Not Assessed Plan A ‡ Plan B ‡ Plan C ‡ Plan D ‡ Plan E ‡ Plan F ‡ Plan G ‡ Plan H ‡ Plan I ‡ Plan J ‡ Plan K ‡ Plan L ‡ Plan M ‡ Plan N ‡ Overall Totals ‡ 1 One or more study indicators demonstrated sustained improvement. ‡ The PIP(s) did not progress to this phase during the review period and/or required an additional measurement period; therefore, sustained improvement could not be assessed.

Common Areas for Improvement Activity I: Study Topic  No historical plan-specific data provided to support the selection of the study topic (Evaluation Element #2)

Common Areas for Improvement Activity IV: Study Population  Length of enrollment required not specified (Evaluation Element #2)

Common Areas for Improvement Activity VI: Data Collection  Timeline for data collection not provided (Evaluation Element #4)  Not all information regarding manual data collection was provided (Evaluation Elements #5-9)

Common Areas for Improvement Activity VIII: Data Analysis  Baseline data and data analysis not reported in this year’s submission (Evaluation Elements #1-5)

Recommendations  Use the PIP Summary Form Completion Instructions when documenting the PIP Summary Form  If you have questions, contact HSAG for technical assistance

HSAG Contacts For any PIP questions or to request PIP technical assistance contact: Christy Hormann Jenny Montano

Questions