Presenter: Pamela Peele, Ph.D. Vice President, Health Economics February 28, 2012 Learning About Your Clinical Programs

One of the nation’s largest Integrated Delivery Systems 5th in NIH funding, affiliated University of Pittsburgh $9.0 billion in total operating revenue More than 54,000 employees More than 3,000 employed physicians and 5,000 affiliated physicians 21 hospitals: 4,500+ beds and 43 regional cancer centers 400 clinical locations; home care; rehab, urgent care 1.8 million members in Insurance Division programs 20,000+ contracted network providers Global and Commercial Enterprise (UK; Italy; Qatar; Ireland; Cyprus) $500 million commitment to information technology UPMC Today 2

2 nd Largest in Nation Provider Led 3 rd Largest Operating in PA 1.8M Members Annual Revenues ~$4B Fastest Growing Medicaid Plan Fastest Growing Children’s Health Highest Commercial Satisfaction J.D. Power Top 10 Nationally in Medicaid Quality 4 Star Medicare Plan Highest Ranked Provider Satisfaction (PA) National Business Group on Health Platinum Winner past three years UPMC Insurance Services Division Highlights 3

1.8 Million covered lives across all insurance products Medicare Advantage Medicare FFS Commercial (fully insured and ASO) Medicaid Special Needs Plan (SNP) Children’s Health Insurance Plan (SCHIP) Community Care (behavioral health management plan) UPMC Health Plan 4

Levels of Analytics Framework 5 Standard Reports What happened? Alerts What actions are needed? Query Drilldown What exactly is the problem? Ad hoc Reports How many, how often, where? Statistical Analysis Why is this happening? Optimization What’s the best that can happen? Predictive Modeling What will happen next? Forecasting What if these trends continue? Degree of Intelligence Competitive Advantage From Tom Farre, “The Analytical Competitor”, in Analytics: The Art and Science of Better, ComputerWorld Technology Briefing. UPMC HP: 2009 UPMC HP: 2006

The TWO Pieces of the Puzzle 6 1.Program Description 2.Analytic Team

1.What are trying to do or change? Concrete objectives of the program with outcomes linked to program actions 2. How will we do it? What are the tasks to be done, who does them, to whom,and when? 3. How will we know if we are successful? Outcomes, hypothesis testing, measurement MAKE AN INTERNAL TEMPLATE DOCUMENT Program Description 7

Pre/Post evaluations –Confounded by unobservables and time-specific changes –Selection criteria often undermines validity Randomized Controlled Trial –Often impractical for regulatory, operational, or member/provider satisfaction reasons Matched cohorts –Requires scientifically sound matching techniques –If continuous enrollment is required, death takes a holiday Avoiding Regression to the Mean 8

Intention to Treat –Requires careful thought on the inclusion/exclusion criteria Differences in Differences –Powerful method for the real world –Eliminates the concern over uniformly distributed unobservables Avoiding Regression to the Mean 9

Pediatric Weight Management Program Objectives 10 A clinic-based program that utilizes a multi-disciplinary team approach to reduce BMI in overweight or obese children Collaborative program with Children’s Community Pediatrics and Children’s Hospital of Pittsburgh Evaluation focused on children with 2 or more weight measures within 182 days (July 2010 to January 2012) and BMI percentile of 85% or greater Research question: Did children enrolled in the program have a greater change in body mass index (BMI) percentile than children not enrolled in the program?

Analysis Design 11 1.Primary analysis: Compared the difference in the change in BMI percentile between the intervention and comparison groups using a matched difference in differences comparison (kernel matching) Matching variables: age, gender, baseline BMI percentile, duration between first and last visit, and the number of visits 2.Robustness checks: Compare results of kernel matching to results using exact matching, nearest neighbor propensity score matching, and two OLS regression models

Primary Findings 12 Difference-in –differences: Δintervention-Δcomparison= =-0.45

Conclusions and limitations 13 Primary conclusions Children enrolled in the Pediatric Weight Management program experienced significantly greater reductions in their BMI percentile compared to children not enrolled in the program These the program and designing a more rigorous study of the outcomepromising findings, if clinically meaningful, provide evidence to warrant continuing s going forward Limitations Program attrition: 86 children (46%) did not have a return program visit Very little information is known about the children in the comparison group

Analysis Population 14

Primary Analysis 15

Robustness checks 16

Levels of Analytics Framework 17 Standard Reports What happened? Alerts What actions are needed? Query Drilldown What exactly is the problem? Ad hoc Reports How many, how often, where? Statistical Analysis Why is this happening? Optimization What’s the best that can happen? Predictive Modeling What will happen next? Forecasting What if these trends continue? Degree of Intelligence Competitive Advantage From Tom Farre, “The Analytical Competitor”, in Analytics: The Art and Science of Better, ComputerWorld Technology Briefing.

Excel Access Crystal Reports Staff Business Analyst (30) Accounting

Current Staff 19 Clinical Program Evaluation (5) Epidemiology Biostatistics Health Services Research Strategic Business Analysis (6) FinanceEconomicsPolicyStatistics Database & Data Quality (6) FinanceEconomicsPolicyStatistics Modeling (3) PhysicsMathematics Biomedical Engineering Statistics Operations (5) Economics Industrial Engineering Operations JournalismStatistics

Industry Knowledge Data visualization skills Data ECTL (extraction, cleaning, transformation, loading) skills Statistics Health Services Research Data Mining Financial modeling & evaluation Presentation, writing, and communication skills Formally trained but NOT blinded by their training –Challenge deeply held beliefs Staff Skills and Backgrounds 20

Database: SQL, Toad Statistics: SAS, STATISTICA, STATA, R Data Mining: STATISTICA, SAS Enterprise Miner, R Modeling & Simulation: MATLAB, Mathematica, Vensim GIS: ArcGIS Tools 21

Data Overload –No Knowledge No Learning Misleading Data Perspective Issues 22

Patient Information 23