E CONOMIC E VALUATION Moderator: Dr. Chetna Maliye Presenter: Rohan

F RAMEWORK Definition The need of Evaluation Responsibility for Evaluation Process of Evaluation Requisites to Programme Evaluation Sequential Steps Methods of Evaluation Assessment of Components Types of Evaluation Programme Evaluation

E VALUATION Definition: The comparative analysis of alternative courses of action in terms of both their costs and consequences' (Drummond, et al., 1997). It is the element-by-element examination of the expenditure to determine how resources have been spent. A critical process needed to solve management problems Helps to understand how funds have been used Guides in optimal utilization of scarce resources

U SES OF E CONOMIC E VALUATION : Accountability Assessing Efficiency Assessing Equity Assessing priorities Making cost projections Considering cost Recovery

E VALUATION PROVIDE BASIS FOR : Determinant the extent of attainment of objectives and progress towards goal Establishing revised standards of performance Identifying strong and weak points/ dimensions/ aspects of programme Challenging, revising/ modifying programme assumptions by testing the implicit rationale of the programmatic intervention Suggesting changes in program operations and objectives Clarifying programme by requiring operational definitions in measurable terms Identifying inconsistencies between programme objectives and activities.

E VALUATION PROVIDE BASIS FOR : Justifying the programme to funding sources and the public Testing alternative approaches and procedures for service delivery Determining the coverage and availability of the programme relating to the existing community needs Ensuring the provision of services which satisfy existing standards Determining the resources required to respond to community needs both in terms of making service available and desired impacts of services provided Comparing the allocation of project resource allocation decisions Identifying the possible side effects of the programme

E CONOMICAL A ND F INANCIAL C OST A NALYSIS VariableFinancial analysisEconomic analysis CostsMonetary cost.Economic (“opportunity”) cost. e.g, actual monetary payments for human resources, materials, or infrastructure e.g, volunteer workers and donated goods Valuation using market interest rates. Valuation using a discount rate OutcomesIncome; expenditure; cash flow; profit; end-of-period balance; internal financial rate of return; net present financial value. Benefit–cost ratio; internal economic rate of return; net present value. Financial consequencesResource consequences

E VALUATION : Evaluation could be divided into Total Partial Time Related Periodicity: Progress and efficiency:-Once a year Effectiveness: In longer time than a year Impact: Longer time at least five years from the inception of the programme

W HO CAN USE COST A NALYSIS Government departments such as ministries NGO’s which largely exists to promote the social good,. Individual or Institutional decision makers such as hospitals. Private enterprises which produce goods where there is demand and potential for profit. Individual and consumers who make purchase decisions based on their perceptions of needs External donors which may include bilateral government agencies, multilateral agencies and international agencies

R EQUISITES TO PROGRAMME E VALUATION : Criteria reasonable to employ before evaluating any Programme:- Programme which have the potential to make an impact on a large number of people. Programme which have a higher potential fro inflicting harm on people. The amount and type of resources required by a specific programme. Pilot or demonstration projects merit a relatively high priority.

Following Programme should not be evaluated: Programme which are too new or too disorganized. Resources for evaluation are inadequate The key administrators are not committed to it or its use.

S TEPS FOR C OST -A NALYSIS Research Question Plan Ecnomic Evaluation study Setting scope of study Deciding on time trend Selecting sample Drawing iternary & checklist Cost estimation Identification and choice of costs Quantification of costs Valuation of costs Adjustment for differential timing Appropriate cost analysis methods Analysis, Interpretation and Presentation Assessment of Impact of uncertainty Impact estimation Identification and choice of impacts Quantification of impacts Valuation of impacts Adjustment for differential timing

M ETHOD O F E VALUATION : Components Assessments of the components and Indicators.

I NDICATORS : Output Indicators: Measure the accomplishments of the programme objectives. Process Indicators: Measure the manner and extent of carrying out the processes under consideration Product Indicator: Measure the outcome of this process

Progress Indicators: Health Policy Indicators : like allocation of resources, community involvement Health status Indicator: like weight of new born, wt/age ratio in children. Social and economic indicators like population growth, GNP, GDP Provision of health care : Availability, Accessibility, utilization of services. Indicators of coverage: Availability of safe water, sanitary facilities, MCH

 Identification of main costs and their sources  Quantification of costs  Valuation of costs and discounting. C OST - E STIMATION

I DENTIFICATION OF COSTS AND THEIR SOURCES Intervention Direct Cost Indirect Costs Wider cost implications to society eg. lost production. Time receiving medical care Wider cost implications to society eg. lost production. Time receiving medical care Non-health services Eg. Patient transportation, Administration Research & Development Non-health services Eg. Patient transportation, Administration Research & Development Health services resource use. Eg. Inpatient, outpatient, tests, drugs Costs to family and friends. Costs to family and friends. Perspective is important Range of costs justified by perspective

S OURCES OF COST DATA Routine Information Survey : ◦ Market prices & Labor statistics Periodic information sources- ◦ contracted company, ◦ government ministry or ◦ NGO Specialist surveys or studies ◦ Household surveys ◦ scientific studies Expert opinion

Q UANTIFICATION OF COST DATA Need to quantify resource use in appropriate physical and natural units ◦ hours, days, miles etc Direct costs are mostly assessed, and categorised as: ◦ Capital costs (buildings, equipment) ◦ Overheads (jointly used resources, such as heating and lighting, administration and catering) ◦ Labour (medical and non-medical staff) ◦ Consumables (disposable items, such as drugs, bandages etc) Need to distinguish between fixed, variable and total cost, and average, marginal costs and incremental cost

F IXED, VARIABLE AND TOTAL COST Total cost: all costs incurred while producing a service Fixed cost - do not vary with quantity in short run e.g. Capital costs Cost Quantity Variable cost : vary with level of service. E.g. consumables

A VERAGE AND MARGINAL COST CURVES Cost Quantity Average cost cost per unit of output Marginal cost cost of producing an extra unit

M ARGINAL VERSUS INCREMENTAL COST Cost Q MC B, Q MC A, Q IC A-B, Q TC of Prog A TC of Prog B Quantity

C OST A NALYSIS M ETHODS 1. Cost-effectiveness analysis (CEA): 2. Cost-utility analysis (CUA) 3. Cost-benefit analysis (CBA): 4. Cost-of-illness analysis: 5. Cost-minimization analysis: 6. Cost-consequence analysis:

C OST -E FFECTIVENESS A NALYSIS A comparison of costs in monetary units with outcomes in quantitative non-monetary units, e.g., reduced mortality or morbidity Target and output indicators pre-decided Most appropriate if the important outcome is uni- dimensional Compare costs of at least two alternatives Cost- Effectiveness Ratio: CE Ratio = ($ Cost Int - $ Cost Comp ) / ($ Effect Int - $ Effect Comp ) For example: “$45,000 per life- year saved” or $10,000 per lung cancer case averted”

T HE COST - EFFECTIVENESS PLANE.

C OST -E FFECTIVENESS A NALYSIS CONT … Advantages 1. Useful to know the cause of failure to attend the objective and the unmet demand. 2. Adequacy of corrective action taken, can be reviewed Disadvantage 1. Only compare programs with similar outcomes 2. There is not enough information to assign a value to the outcome 3. Methodological inconsistency 4. Ambiguity in assessing overall improvement or decrement in health 5. Cannot address the issue of allocative efficiency

C OST -U TILITY A NALYSIS A type of cost-effectiveness analysis It compares costs in monetary units with outcomes in terms of their utility, usually to the patient, measured, e.g., in QALYs Measures of different effects are consolidated into a common abstract scale Example; Quality adjusted life year (QALY), Disablity adjusted life years (DALYs) Most appropriate if the important outcome is multidimensional Disadvantage Difficult to assign utility weights Cost- Utilty Ratio: CU Ratio = ($ Cost Int - $ Cost Comp ) / ($ Utile Int - $ Utile Comp ) For example: “$10,000 per QALY gained”

QALY S GAINED FROM AN INTERVENTION

Q UALITY - ADJUSTED LIFE YEARS (QALY S ) Adjust quantity of life years saved to reflect a valuation of the quality of life If healthy QALY = 1 If unhealthy QALY < 1 QALY can be <0 Procedure Identify possible health states - cover all important/relevant dimensions of QoL Derive utility ‘weights’ for each state Multiply life years (spent in each state) by ‘weight’ for that state. Quality adjustment utility “weights” should (1) be preference-based (2) be interval-scaled (3) contain both perfect health and death on the same scale

T ECHNIQUES TO ‘ WEIGHT ’ UTILITY Direct methods - Measure the preferences of individuals directly using general instruments like the visual analogue scale, the time trade-off method, and the standard gamble technique Widely applied Costly & time consuming Indirect methods Simpler to use (though difficult to develop) Based on multi-attribute health status classification systems.(consist of a set of attributes of health status (e.g., pain and discomfort, visual acuity, ambulation, cognitive function, etc.), and levels of function associated with each attribute from full function to impaired function (e.g. perfect vision — totally blind)

C OST -B ENEFIT A NALYSIS It compares costs and benefits, both of which are quantified in common monetary units Unique feature that can indicate explicitly whether benefits outweigh costs Advantage: Can address allocative efficiency Can compare disaperate technologies Disadvantage: Difficult to assign monetary value to outcomes

C OST -B ENEFIT A NALYSIS CONT … Two approaches: Cost- Benefit Ratio: (Ratio Approach) CB Ratio = ($ Cost Int - $ Cost Comp ) / ($ Benefit Int - $ Benefit Comp ) For example: “Cost Benefit ratio of 1.5” Cost- Benefit Ratio: (Net Benefit Approach) CB Ratio = ($ Cost Int - $ Cost Comp ) - ($ Effect Int - $ Effect Comp ) For example: “Net cost of $15,00”

M ONETARY V ALUATION Earlier “Human Capital’ method were used Willingness To Pay Method Done by Contingency valuation method Assess individual ‘willingness-to-pay’ for (the benefits of) a good through either: 1. Observed wealth-risk trade-off (revealed preference) Advantage – ‘real’ preferences/values Disadvantage – difficult control for confounders 2. Direct survey (stated preference) Advantage – direct valuation of good Disadvantage – hypothetical/survey problems Vast majority of CBA use direct survey

C OST OF I LLNESS A NALYSIS A determination of the economic impact of an illness or condition. Cost of Illness includes: Medical care for prevention, treatment & Social services for rehabilitation Productivity loss Examples o What does cancer cost the India? o What does blindness cost the world? Just because something has the highest cost of illness does not imply that it necessarily should have the most resources directed toward research or cure Depends on how much it will cost to do something about it

C OST - M INIMIZATION A NALYSIS A determination of the least costly among alternative interventions Calculate the cost of the different methods of achieving the objective Needs at least two ways of achieving the objective Example Rabies vaccination: Intramuscular or Intradermal schedule? Which is least costly? If so, should we continue Intramuscular schedule or use intradermal schedule? These studies are difficult because they don’t focus on partial outcomes Need a high degree of certainty that outcome can be obtained or else these studies are not particularly helpful

S UMMARY OF C OST - A NALYSIS M ETHODS Valuation of costs (in Monetary units) Valuation of outcomes Comments Cost Effectiveness $÷Natural units It can only compare technologies whose outcomes are measured in the same units Cost Benefit$ ÷ or – i.e, Ratio Approach or Net Benefit Approach $ It enables comparison of disparate technologies Difficult to assign monetary values to all pertinent outcomes Cost Utility$÷ Utilities (e.g., QALYs) It enables comparisons of disparate technologies Cost of Illness$vs.None A determination of the economic impact of an illness or condition. Cost Minimization $vs. Assume same outcome A determination of the least costly among alternative interventions

D EALING WITH UNCERTAINTY Sensitivity analysis ◦ Systematically examining the influence of uncertainties in the variables and assumptions employed on the estimated results Steps: 1. Identifying the (uncertain) variables ◦ All variables in the analysis are potential candidates ◦ Give reasons for exclusion rather than inclusion 2. Specifying the plausible range over which they should vary ◦ Reviewing the literature ◦ Consulting expert opinion 3. Recalculating results based on combinations of the best guesses, most and least conservative, usually based on… ◦ One-way analysis (each variable separately) ◦ Multi-way analysis (number of variables together) ◦ Threshold analysis (amount of variance needed to achieve specified result)

A NALYSIS, INTERPRETATION & PRESENTATION Set up "dummy tables" and graphs at initial phase Analysis should include interpretation- i.e. What do the data mean? Finally, Report the result Stick to objective Keep presentation simple and use tables & graphs Write an executive summary

S TRENGTHS & L IMITATIONS Strengths o a very useful tool for the manager and policy maker o In the absence of certainty, even approximations can help improve decision-making Limitations o It's difficult to estimate "true costs" or opportunity cost o Allocation of costs; often very difficult

R EFERENCES : Creese A, Parker D. Cost Analysis in Primary Health Care. A training manual for programme managers. Geneva: World Health Organization; Torres T, Baltussen R, Adam T, Hutubessy R, Acharya A, EvansD, Murray C. WHO Guide to cost effectiveness analysis. Geneva: World Health Organization; Hutton G, Rehfuess E. Guidelines for conducting cost–benefit analysis of household energy and health interventions. Geneva: World Health Organization; Module E Cost and Sustainability analysis. Aga Khan Foundation Cost Analysis Methods. Available from URL:

Thank You