Articulating the Value of Consumer Engagement 2 – 3:30 pm ET July 25, 2013

Context Context CE Sustainability RFP: 8 projects Technical assistance available o Webinar series o Individual calls Contribution to the field Other Sustainability TA underway

Articulating the Value of Consumer Engagement Carla Zema, PhD Zema Consulting July 18, 2013

Outline Importance Recap of ROI from Sustainability Workshop Articulating Value – Understanding Different Methods Review of Key Elements of Analysis Determining Costs and Benefits Valuing Benefits Conclusion and Next Steps

Importance One of the great mistakes is to judge policies and programs by their intentions rather than their results -- Milton Friedman

More Effective Less Effective More Costly Less Costly ADOPT DO NOT ADOPT Tradeoff to consider Weighing Effectiveness and Cost

Use of results Justify implementation to internal and/or external stakeholders Interest potential partners, such as funders, community partners Justify use of resources Complement other evaluations

ROI Template ElementContents DescriptionScope of the activity or program Audience/point of viewStakeholders/funder(s) who will obtain ROI from participation Financial benefitsHard dollar categories, $ for each Strategic benefitsDescription with $ if possible Intangible benefitsDescription, qualitative value CostsHard dollar costs; other costs a stakeholder might want to include Additional componentse.g., time, probability of success ROICalculation and/or qualitative evaluation AF4Q Sustainability Workshop, 2013

A rose by any other name… ROI Cost-benefit analysis (CBA) Cost-effectiveness analysis (CEA)

Comparison of Methods CEACBAROI Basic calculati on Cost-effectiveness ratio = Total costs Measure of effectiveness Net benefit = Total benefit-Total costs Ratio = Total benefits/Total costs ROI = Net benefit Total cost ResultCost per measure of effectiveness (or incremental/margin al cost per unit measure of effectiveness) Net benefit valued in dollars Ratio results in number greater than or less than 1 Percentage

Comparison of Methods CEACBAROI Pros Easy to compare programs with similar outcomes Useful if outcomes are difficult to value Results are in dollars making interpretation easy Useful when considering only one program and/or when programs have different outcomes Easy interpretation “ROI” more widely known Cons Measure of effectiveness can be awkward to interpret Difficult to use if more than one benefit Can be difficult to value benefits Indirect or intangible benefits are often not included

Steps to Conduct an Analysis Define the framework for the analysis Determine the perspective of the analysis Decide on the timeframe for the analysis Define the population Identify and measure the costs and benefits Value the benefits (for a CBA) Discount (depending on timeframe) Calculate results Conduct a sensitivity analysis Use results for decision-making

Define the Framework What is the overall objective? Which options are being considered? How will effectiveness be measured? What is the status quo? (What if no action is taken?) Will the analysis be prospective, retrospective, or concurrent? Will a CEA or CBA be conducted?

Determine the Perspective of the Analysis Critical to defining the costs and benefits Must be maintained throughout the analysis May or may not be the primary audience for the results Examples: o Funders o Payers o Practice o Employers o Societal

Decide on Timeframe Must be clearly defined Considerations: o Does program have a specific timeframe for implementation? o When would results be expected? o What timeframe is meaningful to the primary audience for the results?

Define the Population Considerations: o Particular subgroup of patients or practices? o Certain geography (e.g. state, county)? o Types of practices? o Multiple systems? o Types of payers? o All practices or just those with a certain number of patients/provider?

Identify and Measure the Costs and Benefits Categories: o Direct o Indirect o Intangible Include only actual or real costs and benefits that are beyond the status quo Do not include transfers Handling missing data

Value the Benefits for a CBA or ROI For a CBA or ROI, benefits must be valued Can be “gains” or “costs avoided” Options: o Market value of benefit o Downstream financial impact of the benefit Make sure the valuations are realistic and meaningful to stakeholders

Discounting Costs and benefits should be discounted if greater than one year No consensus on standard discount rate Range of 3%-5% with 4% commonly used for social program

Calculate Results Spreadsheet software such as Microsoft Excel make calculations simple and transparent Keep each costs and benefit as well as quantities and rates as separate cells to increase transparency and make changes easier Results cells are then simply functions that calculate the results

Conduct a Sensitivity Analysis Opportunity to explore the variation in the results if inputs are modified Helpful given that some inputs have some uncertainty or a range of possible values Two types: o Partial o Extreme case

Partial Sensitivity Analysis One input variable is varied at a time while holding all other values constant Compare new results with base case results Consider varying variables: o representing assumptions that were made for the analysis o that have a range of possible values o that were estimated due to missing or unavailable data o that have uncertain values

Tornado Diagram Example Payer mix Reimbursement rate P4P incentive Increase in improvement Number of providers Number of practice staff Practice volume Cost of training Number of patient partners

Extreme Case Sensitivity Analysis Similar to partial sensitivity analysis in terms of variables to vary in the analysis Instead of one variable at a time, all variables that are selected are varying simultaneously o Once in a “best-case” scenario o Again in a “worst-case” scenario

Summary CEAs/CBAs are not a silver bullet and do not provide a definitive answer Offer important information to inform decision-making and complement other evaluation information on effectiveness Critical component of any program or project evaluation

Next Steps Webinar series will walk you through the process Homework : o Finalize framework o Begin to identify costs and benefits Individual technical assistance is available Next webinar will be opportunity to get group feedback

Contact Information Carla Zema, PhD Zema Consulting (724)