1. Economic Evaluation in D & I Research
Tzeyu Michaud, PhD
Department of Health Promotion, Social, & Behavioral Health
UNMC

2. Overview What is economic evaluations?
Why conduct economic evaluations?
Review of economic evaluations
How to start an economic evaluation?
Return on Investment
Example
Q & A session
Today’s talk is arranged in the follow orders.
I will talk about what is economic evaluations? why we care about them? and I will review the types of the economic evolutions and then I will talk about a step-by-step process to start an economic evaluation study. And I will end the talk with some examples to show you how this approach could be applied.

3. What is Economic Evaluations?
A tool to make comparisons of alternative options in terms of their costs and consequences
It provides a systematic way to identify, measure, value, and compare the costs and consequences of various programs, policies, or interventions
It is comparative
1. So, what is economic evaluation? by definition, it is a tool to make comparisons of alternative options in terms of their costs and consequences.
2. Alternative options here refer to the range of ways in which health care resources can be used to increase population health, such as pharmaceutical or surgical interventions, screening and health promotion programs.
3. Consequences refer to all the effects of health care programs and generally focus on changes in individuals’ health, which can be positive or negative, but also can include intangible effects that individuals may value, such as reassure and information provision.
3. In other words, economic evaluation approaches provide a systematic way to identify, measure, value, and compare the costs and consequences of various programs, policies, or interventions.
4. So, per the definition, we can say economic evaluation is comparative. That is to say, it is not possible to establish a full economic evaluation of an intervention unless its cost and consequences are compared with at least one alternative option. r

4. Why conduct economic evaluations?
To inform the decisions of various health care systems about which health care interventions to fund from available resource
To evaluate whether an intervention are worth implementation and
To evaluate whether the benefits from already implemented interventions have been worth the costs
To ensure society can receive a good return on its investment in public health
So why we care about it? We know that all public health programs, policies, or interventions have costs or financial investments, and the reality is that our resources are limited and they can be used for many different purposes.
Under this circumstance, decision makers, policy makers or practitioners need information about the cost and the effectiveness of potential strategies to ensure that good value is obtained for the resources invested in such strategies. In other world, a good return on society’s investments in public health by implementing a program, carrying out an intervention, or a policy.
Accordingly, economic evaluation can be used to
inform the decisions of various health care systems about which health care interventions to fund from available resource, such as decisions about the coverage or reimbursement of new pharmaceuticals. For example, In UK, the national institute for health and clinical excellence (NICE) uses economic evaluation to inform decisions about medical device, diagnostic technologies and surgical procedures, as well as pharmaceuticals.
aslo, Decision-makers want to know whether an intervention are worth implementation and whether the benefits from already implemented interventions have been worth the costs.
And finally, the ultimately goal of economic evaluation in public health is for society to receive a good return on its investment.

5. Healthcare Evaluation
Are both costs and outcomes measured? No
Yes
Examine outcomes only
Examines costs only
cost-outcome description
outcome description
Cost description
Efficacy or effectiveness evaluation
Cost analysis
Full economic evaluation
Cost-effectiveness analysis
Cost-benefit analysis
Cost-utility analysis
Cost-minimization analysis
Before we review the different types of economic evaluations, we should talk about healthcare evaluation first. Considering a 2-by-2 table, the top dimension looking at whether both costs and outcomes are included in the evaluation, and the other one looking at whether more than two alternative options are included in the evaluation. Based on the answers to these two questions, a six-cell matrix for evaluation situations is defined.
For studies without comparison groups and only have either costs or outcomes data, or both, the evaluation is labelled a cost-description, outcome-description, or cos-outcome description.
The most basic type of economic evaluation is a cost analysis, where at least two alternative options are included with only has cost data in the evaluation. A cost analysis can be used when the effectiveness of an intervention is not yet known. For instance, if we are evaluating a pilot project that lasts two years, but the impact on health outcomes is not expected until several years beyond that, then we may want to have information on cost of reaching intermediate targets. Also, a cost analysis is useful for comparing two interventions where the effectiveness is not the same.
For studies only fulfill one of the conditions, they are considered as partial evaluation relative to the full evaluation, where both cost and outcomes are measured and at least two services or programs are being evaluated. That is our focus today.
Compare two or more alternatives?
Source: Drummond et al. Methods for the Economic Evaluation of Health Care Programmes, 2005.

6. Type of (full) Economic Evaluations
Method
Costs
Effects
Evaluation Question
Cost-effectiveness analysis
Monetary unit
Natural units (life-years gained, disease events prevented)
Comparison of intervention with same objectives
Cost-utility analysis
Utility, QALYs, or DALYs
Comparison of intervention with different objectives
Cost-benefit analysis
Are the benefits worth the costs?
Cost-minimization analysis
Effects are not measured (they are considered to be equal)
Least-cost comparison of programs with the same outcomes
There are four types of full economic evaluation of interventions: cost effectiveness analysis, cost-utility analysis, cost-benefit analysis, and the cost-minimization analysis. Here we summarize the characteristics of these four analyses. The difference are originated from the program effects measured in the intervention programs and the evaluation question we are interested.
In cost-effectiveness analysis, the effects of interventions are presented in naturally occurring units, such as deaths, disease prevented, and compared with other interventions that serve the same goal .The main advantage of cost–effectiveness analysis is that measuring benefits in natural units simplifies the analysis and is often more intuitive for users of the study.
On the contrary, cost-utility analysis can be used to compare interventions with different objectives, because the effects of the interventions are presented as utility. Therefore it can be used to decide the best way of spending a given treatment budget or the health care budget as a whole. The most commonly used units of utility are quality-adjusted life-years (QALY) and disability-adjusted life-years (DALY).
In the cost-benefit analysis, both costs and benefits of interventions are expressed in the monetary units. The main practical problem with cost–benefit analysis is valuating benefits, such as saving lives or relieving pain, in money units.
4. Finally, the cost-minimization analysis, are similar to cost analysis, since no effect data is required but with the premise that the benefits of the interventions are the same. This analysis is rarely an appropriate method also due to this limitation.
So which analysis to choose? It depends on what data do you have or you plan to collect.
Adapted from WHO methodological approaches for CE and CUA of injury prevention measures, 2011.

7. How to Start an Economic Evaluation?
Here I am going to show you the flow chart which may help determine which type of approach/analysis is most appropriate and feasible.
Starting from the study design section, we need to identify the study question and objectives, identify and describe the alternatives. And next we need to choose a perspective, which decides which costs should be considered in evaluation. and then we decide the time horizon of the evaluation, which allowed us to capture either the short-term or long-term costs and effects, for the analysis.
After we have our study design at hand, next we would look at what data we have to conduct the economic evaluation. Since this is a economic evaluation study, we will start from asking a cost data. Do we have cost data? if we have cost data then the next question to ask is whether we have outcomes data, if the answer is no, then we will conduct a cost analysis, if the answer is yes, we can choose one of 4 types of analyses, that we just talked about, based on our research questions and effect units. If we do not have cost data , then we need to plan a data collection process in our study.
Here, we categorize costs into two groups- intervention costs and additional costs. What to include or not include in the intervention costs depend on the research question. If the intervention is a drug treatment, only the costs of drug and may be the follow-up physician visits will be include as the intervention costs. On the other hand, if the intervention is a community weight loss program, there would be more components need to be included.
Here, cost data are roughly categorized into two groups- intervention costs, and additional costs. as for which costs could be included in these groups, I will show you in the following slides.
On the next few slides, we will talk about cost data collection, what to consider and account for in the analysis.
step 1. define the problem and objectives of the study
step 2. identify the alternatives
step 3. describe each alternative
step 4. define the perspective of the CEA
next identifying and collecting cost effectiveness data
step 5. identify, measure, and value costs
step 6. identify and measure effectiveness
step 7.

8. Intervention Costs Fixed costs Variable costs Equipment/ technology
Facilities (e.g. office space, or intervention space)
Other
Variable costs
Personnel
Training
Administrative/ operating
Other (e.g. supplies)
So from a broader perspective, we can further separate intervention costs to fixed costs and variable costs. Fixed cost may include equipment or technology used to deliver the intervention. facility includes office space or intervention space. In some fields, people may call this category the overhead costs. Variable costs may include personnel, such as care coordinators, healthcare coaches, or facilitators. training includes staff training session and training materials in order to deliver the intervention. Other costs may include administrative, operating , maintenance, and others. So, if you plan to conduct a economic evaluation for a intervention program, please incorporate the cost data collection step in the study.

9. Which costs are counted?
cost components/perspective
Societal
Healthcare sector
Payers
Formal healthcare
paid by payers
Yes
out-of-pocket No
Informal healthcare (e.g. transportation, unpaid caregiving time, or patient time)
Non-healthcare (e.g. loss of productivity
So in addition to the intervention costs, what else should be considered in the economic evaluations? The answer depends on which perspective we are taking on for the evaluation. In public health, cost is usually determined from the perspective of society as a whole. However, most of time we choose a perspective based on the question of who is going to invest/pay for the intervention program. Therefore, cost component can be separated into formal healthcare, which indicates the costs that are directly associated with a healthcare service, which either paid by the payers or paid out-of-pocket by patients. Informal healthcare indicate the costs that is indirectly associated with a healthcare service, such as the time traveling to the clinic or the unpaid care giving from the family members, or the time patients use to see a doctor. Non-healthcare includes loss of productivity due to the illness.
The take-home message from this chart is that the perspective taken in an economic analysis can have an important influence in how an intervention is assessed and the results obtained and interpreted.
n CEA, the term indirect costs refer to productivity loss, which are costs related to
The panel tries to avoid using the term of indirect
direct medical costs (formal costs), also can be called formal medical costs, paid by payers and paid by patients out of pocket.
Direct non-medical cost (informal healthcare)
Direct healthcare costs (the costs of test, drugs, supplies, healthcare personnels), vs. direct non-healthcare costs (chid care costs for parent undergoing treatment or the costs of transportation to and from a physician’ offices, or the costs of a family member spends caring for a viable relative)
non-healthcare (indirect)
Health care costs refers to the tangible resources available to the health care system. for example, clinical and other staff, capital equipment and buildings, and consumables (such as drugs).
Non-health service resources are also used to produce health care, such as the time of patients and their families.
Adapted from Cost-Effectiveness in Health and Medicine, 2017.

10. Source of Cost Data Collection
Administrative data base
Survey for providers and beneficiaries
Observational studies
Expert panel
Published price list (e.g. average wholesale price)
Published literature
Others
After we decide which cost components to include in the analysis, the next question is what if we don’t have some of the key data component at hand and we are unable to collect them from the study. Here we provide some data collection sources. We can either running the data analysis using the administrative database or observation studies to obtain the cost data that fit into our study. Or we can derive this information from the literature review if there is not much variation of the components across populations. Expert panel approach is usually used when there is no price for a new treatment, or an innovative dialogistic tool.

11. $ Gained (- $ Cost of Investment)
Return on Investment
$ Gained (- $ Cost of Investment)
ROI =
Now I am going to briefly talk about the concept of return on investment, as requested by Paul.
Return on investment is a concept from personal investing where the intent is to maximize gain (or money earned on an investment) in relation to the cost of the investment. The gain or net gain is divided by the cost of the investment to calculate the return on investment. The formula could be written in two ways with different numerators- one purely includes gained from a investment, the other includes the net gained after subtracting the costs of investment.
ROI is often used in marketing different types of financial investments. However, we need to be cautious when interpreting ROI because the calculation can be easily changed by manipulating what counts as a “gain” and what counts as a “cost”, or based on what perspective.
In addition, we can view all types of economic evaluations as a return on investment analysis since the ultimately goal of economic evaluation in public health is for society to receive a good return on its investment.
In the next few slides, we are going to show you some applications of economic evaluations of a health promotion program.
$ Costs of Investment
Depends on what to include in gained and costs.

12. Weight and Win: a 12-month community weight loss program
Implementation costs millions
Program evaluation
Reach: 33,656 person who were overweight or obese (African American were over-represented)
Effectiveness : 47% and 34% of participants loss 3% and 5% of initial body weight, respectively
Cost per clinically meaningful weight loss for African Americans ($258/ 3%loss; $336/5%loss) was lower than that for Hispanics ($319; $431) and Caucasians($314; $442)
Example 1:
Weigh and Win, (implemented in Denver, Colorado), is a 12-month community weight loss program with daily and/or text support, online access to health coaches, objective weight assessment and modest financial incentives to increase program reach. Participants were recruited through 83 community-based kiosks in denver Colorado between 2011 and 2014.
The program was evaluated in term of reach, representative, effectiveness, and cost
2. In the dimension of reach and representative, over four years, 33,656 persons who were at least 18 years old and with BMI ≥ 25 enrolled in the program. Among them, African Americans were over-represented in the sample based on the census data.
3. In the effectiveness dimension, among participants who engaged in the program beyond an initial weigh-in, which is about 20 thousand participants, 47% or 34% lost 3% or 5% of their initial body weight, respectively. Among them, again, African American participants were more likely to achieve 5% weight loss (25%) compared to White and Hispanic participants (19%) (20%).
4. in the cost dimension, cost per clinically meaningful weight loss for African Americans was lower than that for Hispanics and Caucasians.
Based on the these results, we can conclude that Weight and Win is a very effective weight loss program based on the RE-AIM framework. However, is this a good return on the investment of 2.82 millions dollars. did the benefits from the Weight and Win worth the costs to attract potential stockholders to invest such programs?
To answer this question, we further conduct a full economic evaluation for this health promotion program.

13. A state transition Markov model to project lifetime economic outcome and the degree of disease averted, compared with no intervention
The program was predicted to avert (with a corresponding estimated medical costs saved of)
78 cases of coronary heart disease ($28 million)
9 cases of strokes ($971,832)
92 cases of diabetes ($24 million)
3 cases of breast cancer ($483,259)
1 case of colorectal cancer ($357,022)
The estimated medical costs saved per participant was $1,403 ($1,077 of African American men and $1,532 of Hispanic men), and the ROI was $16.7 ($12.8 for African American men and $18.3 for Hispanic men)
In this economic evaluation study, we evaluated the lifetime health and economic consequences of this community weight loss program aiming for overweight and obese adults.
Methods: We applied a Markov model simulation approach to project lifetime economic outcome and the degree of disease averted as a result of a weight loss intervention, compared with no intervention, from a payer perspective. We obtained the Effect sizes of the intervention from Weight and Win and other data from the literature.
Results: Simulation results showed that among 33 thousand participants, the program was predicted to avert 78 cases of CHD, 9 strokes, 92 cases of diabetes, 1 case of colorectal cancer, and 3 cases of breast cancer over the participant lifetime. At a cost of 2.82 million dollars of the program, the estimated medical costs saved per participant was $1,403 ($1,077 of African American men and $1,532 of Hispanic men), and the ROI was $16.7 ($12.8 for African American men and $18.3 for Hispanic men) over the lifetime course of overweight and obese participants for every $1 invested.
With these results, now we can say the weight and win program is a cost-effective approach to promote weight loss in the community setting. and this information may be useful for future adoption and dissemination project.

14. Cost-Effectiveness Analysis
Now let’s move on the next Example.
The US preventive Services task Force in 2009 recommend increased aspirin use for primary prevention of cardiovascular disease (in men ages 45 to 79 years and women ages 55 to 79 years for whom benefit outweighs risk. So, in this study, we evaluated the cost-effectiveness of a statewide public and health professional awareness campaign to increase regular aspirin use among this target population in Minnesota to reduce first CVD events.
Similarly, we applied the Markov model simulation approach to estimate the quality-adjusted life year, costs, and the number of CVD events averted among the target population without a prior CVD history, from a payer’s perspective. The results showed the with this statewide education campaign, it was predicted to avert about 10 thousand primary myocardial infarction in men and about 12 hundred primary ischemic strokes in women, compared to no campaign. The results further indicated this campaign is cost-saving, which mean the campaign generated more benefits and cost less due to averted CVD events than no campaigns.

15. Cost Components
Here is the cost component included in this study. we calculated the campaign cost in per-person basis from a pilot study. The rest of the cost data were derived from the literature review and expert panel. As we emphasize in the previous slide, the perspective we take for the economic evaluation study would determine which component to be include in the analysis, and it in turn may have a significant impact on the conclusion of whether the intervention program is cost-effective or not. Before we start the economic evaluation study, we should think who is our audience/readers and therefore, what kind of cost data we should collect, either primary, or secondary.

16. Summary Economic evaluation is comparative
Plan ahead (cost data, alternative comparison, perspective, outcomes measured)
ASK for help!
The take-home message is
remember economic evaluation is comparative, so think what is your comparison group. which program you are comparing to. status quo could be a comparison group.
Plan ahead, especially about the data collection.
Finally, please do hesitate to ask for help, most importantly in the study planning process.

17. Resource Cost-Effectiveness in Health and Medicine, Gold et al., 1996
Methods for the Economic Evaluation of Health Care Programmes, Drummond et al., 2015
Guidelines for reporting economic evaluations for health interventions. content/uploads/2013/04/Revised-CHEERS-Checklist- Oct13.pdf
Tufts CEA registry. hingtheCEARegistry/SearchtheCEARegistry.aspx
Here I list some useful sources/ materials for conducting an economic evaluation study.
This third one is a checklist provided by the annals of internal medicine for authors who submit a economic evaluation study to their journal. It would also be a good checklist if you plan to conduct an economic evaluation for study.
The final one is a cost-effectiveness analysis registry. It is also very resourceful.

18. Thank You

19. Reported Outcomes

20. Cost-Effectiveness Analysis
Relevant when alternative options have different costs and health consequences
Costs are valued in monetary terms (e.g., US$)
Benefits are valued in terms of clinical outcomes (e.g., cases prevented or cured, lives saved, years of life gained, quality-adjusted life years gained)
Results reported as a cost-effectiveness ratio

21. Cost-Effectiveness Ratio
Net increase in health care costs
Net gain in health effects
quality-adjusted life years
cost-utility analysis

22. Example 1

23. Example 3