1. Cairns 16th October 2014 Professor Christopher Doran
Insights from ACE Prevention: what worked and what needs to be done in economic evaluation
Cairns 16th October 2014
Professor Christopher Doran

2. Overview Introduction to economic evaluation
Overview of ACE Prevention
Introduction to impact assessment
2 Indigenous examples of impact assessment

3. Context of economic evaluation
Resources are scarce in relation to needs / demand
Scarcity forces choices to be made and choices imply a sacrifice or foregone opportunity

4. Economic evaluation is …
The comparative analysis of alternative courses of action in terms of both their costs and consequences in order to assist policy decisions”.
Program A
CostA
ConsequencesA
Choice
Program B
CostB
ConsequencesB

5. Types of economic evaluation

6. Assessing Cost-Effectiveness-Prevention
Rob Carter, Theo Vos, Chris Doran,
Alan Lopez, Andrew Wilson, Ian Anderson, Jan Barendregt, Wayne Hall

7. Assessing Cost-Effectiveness (ACE) studies in Australia
Pilot project in cancer prevention (2000)
ACE–Heart Disease ( )
20 + interventions for prevention of coronary heart disease
ACE–Mental Health ( )
20 + interventions for depression, schizophrenia, anxiety and ADHD
ACE-Obesity ( )
Focus on childhood interventions
ACE-Alcohol ( )
Around 10 interventions to reduce harm from alcohol misuse
ACE-Prevention ( )
ACE-Alcohol Indigenous ( )

8. ACE Prevention: methods
Understand natural history of disease (from burden of disease study)
Analyse current practice: % receiving intervention(s); adherence
Efficacy/effectiveness from literature
Impact in routine Australian health services?
Model change in health outcomes (often over a lifetime) in DALYs
Difference in costs of intervention & cost offsets
Cost-effectiveness ratios in $$/DALY
Mix of most cost-effective interventions

9. From policy to measurement of benefit
Two-stage approach adopted in ACE
First, a measure of health gain in relation to resources consumed ($ cost per DALY)
Picks up element of cost, efficacy/effectiveness and efficiency objectives
Second, explicitly provide for broader considerations not in this C/E ratio
Which we call our ‘2nd stage filters’ (equity; acceptability; feasibility; size of the problem)
Plus confidence in evidence base

10. Presenting cost-effectiveness
Can put all costs (y-axis) and health effects (x-axis) on a graph
Slope of the line represents the economic attractiveness of an intervention
costs
Slope = CER =
health effects
The flatter the slope, the more cost-effective

11. Topic areas and interventions
Total population
Indigenous
 Topic
Prevention
Treatment
Alcohol 9 2
Tobacco 8
Physical activity 6
Nutrition 26
Body mass
Blood pressure/cholesterol 12 5
Bone mineral density 3
Illicit drugs 1
Cancer
Diabetes 7
Renal disease 4
Mental disorders 11 10
Cardiovascular disease
Other 18
Total
123 27 19

12. Intervention pathways: ‘Ideal mix’

13. Alcohol intervention pathway

14. Combining everything in one model

15. Combined impact 43 very cost-effective prevention measures
1 million healthy life years
$4 billion
upfront
investment
Immediate cost savings in blood pressure & cholesterol
Treatment cost saved

16. ACE Prevention - main findings
Areas amenable to preventive interventions to reduce size of burden :
Substantial
CVD, diabetes, kidney disease, tobacco, alcohol, physical inactivity, salt
Moderate
Mental disorders, drug use, osteoporosis
Modest to small
Obesity (unless regs/tax work), F&V, cancer screening

17. ACE Prevention - summary
Pros
Good engagement with policy makers / Indigenous leaders
Platform of recent epidemiological data
Used sophisticated methods
Attempted to consider equity, acceptability, feasibility
Very good dissemination and capacity building (eg. PhDs)
Cons
Very technical – policy makers found it hard to understand, eg. What is a DALY?
Focus on health outcomes – for certain risk factors (alcohol) non-health very important
Relied on secondary data of mixed quality
Modelling considered a black box – not very researcher friendly

18. ACE Prevention – what next?
Funding stopped
limited interest / funding to extend methods in Australia
Centre for Burden of Disease and Cost-Effectiveness ceased to exist
Majority of staff left UQ
Prof Lopez moved to Uni Melbourne
Prof Theo Vos and A/Prof Lim moved to Uni Washington to work on the Global Burden of Disease study funded by Bill Gates
Others now working at World Health Organisation, Oxford University + other Australian Unis
I relocated to Hunter Medical Research Institute to focus on translation research and impact assessment

19. The imperative for measuring
research impact…
… there is a need to maximise the translation of effective research outcomes into health policy, programs and services
The generation and use of high-quality, relevant research evidence will improve health policy and program effectiveness, achieve better health and help build efficient services.
Wills Review, 2012
In Australia the debate on improving health outcomes has relied too much on arguments about increasing resources, and not enough on improving productivity and effectiveness through microeconomic reform and translation of innovations from research.
Mckeon Review, 2013
… this need is central to the Wills and McKeon reviews

20. The imperative for measuring
research impact…
Socially responsible and good for patients
Policy makers and the community are looking for research that is likely to provide a positive social return on investment (SROI)
Policy is already changing
Reward research that demonstrates its potential (and actual) ‘research translation’
Onus on researchers to demonstrate ‘value for money’
Further evidence of this shift in policy
NHMRC (NHMRC Advanced Health Research and Translation Centres)
ARC (principles of research translation).

21. Existing work in this field
Measuring research impact
Payback method: Buxton& colleagues UK in 1996
Core domains of benefit, each with metrics: knowledge, research, political and administrative, health sector and economic. Scores to represent success in each domain
Other versions: Canadian Institutes of Health (2005), Research Impact Framework (2006), Canadian Academy of Health Sciences Framework (2009)
Becker list (Washington University School of Medicine) (Last update 2014)
All include a dimension of economic impact.
AU Government
NSW Government Evaluation Framework (2013)
Cooperative Research Centre (Impact Tool)

22. Translational research pathway (From an economic perspective)
Demand for the research
Program aims
Activity (i.e. what will the research do?)
Outcomes (i.e. what will the research ‘produce’)
Use of the outcomes in the community
Impact or benefit (i.e. how does the community benefit from the research outcomes )
COST
COST
BENEFIT

23. Example: Family well-being (FWB)
FWB program focuses on the empowerment and personal development of Indigenous people through people sharing their stories, discussing relationships, and identifying goals for the future.
Workshops are held with both adults and children to highlight the various health and social issues experienced by Indigenous communities and the steps that can be implemented to deal with these issues.
HMRI are working with James Cook University to identify the economic impact of the program of the program

24. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Deliver course content, refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, the Yaba Bimbie Men’s Group, Yarrabah, Gindaja Treatment and Healing Service, Yarrabah, Ma’Ddaimba-Balas Mens’s Group, Innisfail and Queensland Department of Com
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

25. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Deliver course content, refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, the Yaba Bimbie Men’s Group, Yarrabah, Gindaja Treatment and Healing Service, Yarrabah, Ma’Ddaimba-Balas Mens’s Group, Innisfail and Queensland Department of Com
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

26. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, the Yaba Bimbie Men’s Group, Yarrabah, Gindaja Treatment and Healing Service, Yarrabah, Ma’Ddaimba-Balas Mens’s Group, Innisfail and Queensland Department of Com
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

27. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, the Yaba Bimbie Men’s Group, Yarrabah, Gindaja Treatment and Healing Service, Yarrabah, Ma’Ddaimba-Balas Mens’s Group, Innisfail and Queensland Department of Com
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

28. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, Gindaja Treatment and Healing Service, Yarrabah etc. Is there a cost to deliver this program?
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

29. Demand for the research Use of the outcomes in the community
Translational research pathway
(An example from Family Wellbeing - FWB)
Demand for the research
Demand for the FWB program is mostly generated by Indigenous groups and organisations. The program targets participants who have experienced, or are experiencing, distress and/or trauma in their lives.
Program aims
to provide life-skills for performing leadership roles and dealing with difficulties of life. skills include : Leadership; Basic human needs; Relationships; Life journey; Conflict resolution; Emotions; Crises; Beliefs and attitudes; and, Sensitivity as a leader.
Activity (i.e. what will the research do?)
Refine content to specific groups as needed, provide post-course support.
Outcomes (i.e. what will the research ‘produce’) empowerment; better lifestyle choices; ability to cope with conflict; better engagement with family
Use of the outcomes in the community
RFDS, Gindaja Treatment and Healing Service, Yarrabah etc. Is there a cost to deliver this program?
Impact or benefit: Reduced risks for disease & injury, better health (cost avoided), reduced violence (cost avoided), higher wellbeing (GEM), increased capacity
COST
COST
BENEFIT

30. COST AIMS & ACTIVITIES OUTCOMES IMPACT DEMAND
DEMAND: E.G. A need to enhance wellbeing
Funding to address the "need"
Develop custom FWB program to address the specific "need"
Deliver custom FWB program
Empower individuals through life-skill training
Better lifestyle choices: self management of health, diet, exercise, smoking
Reduced risk factors for disease & injury
Better health outcomes; improved life expectancy better quality of life / wellbeing
Better able to cope with conflict
Reduction of stress; reduction in domestic violence; better quality of life (wellbeing)
Better engagement with family, community, and others
Improved quality of life (wellbeing)
Development of personal responsibility and improved leadership skills
Better able to lead self / familiy / community
Improved wellbeing for self and others
Learning
Increase capacity of self / community and organisation
Employment
Increased income; sense of control, wellbeing

31. Developing a framework to evaluate the impact of Family well-being?
Our framework includes a specific FWB survey that enables us to collect information pertinent to assessing impact
If the evaluation is conducted as a prospective exercise (rather than retrospective – as occurs in most cases) it can also provide ongoing feedback to researchers / service providers on performance.
In this way it can act as a component/facilitator of continuous quality improvement
We acknowledge some problems but we are working on this
The GEM is an appropriate measure of wellbeing
Currently cannot convert changes in wellbeing scores to $ values
Some international work is suggesting that wellbeing be included in all cost benefit analyses and the UK Treasury have published a paper on their attempt to convert wellbeing into $ values
Our aim is to advance this research in Australia

32. Example: Institute of Urban Indigenous Health
The Institute of Urban Indigenous Health (IUIH) was established in July 2009 as a strategic response to the growth and geographic dispersion of the Aboriginal and Torres Strait Islander population in South East Queensland (SEQ) which accounts for 38% of Queensland’s, and 10% of Australia’s total Aboriginal and Torres Strait Islander population.
The role of the IUIH is to lead health service planning, develop and co-ordinate health service delivery, and to play a major role in the development of partnerships between health care providers
The IUIH activities are diverse, multifaceted and lead to a range of outputs. The impacts of these activities are closely aligned with the strategic goals of the IUIH: to improve access to comprehensive primary health care; to develop an effective and culturally aware workforce; to build sustainable partnerships; and to contribute to building the evidence base. While some of these impacts may lead to quantifiable economic return, others are more difficult to quantify. In this analysis benefits were quantified for avoided hospitalisations, avoided time in hospital and the economy wide benefits from employment.

33. IMPACT / BENEFIT OUTPUT / OUTCOME COST ACTIVITY IUIH budget
Service delivery
Preventative health
Workforce development
Service delivery (health assessment, chronic care plan and review)
Preventative health (deadly choices, tobacco action, community events, educational programs)
Workforce development (student placements, staff recruitment, work it out)
Sustainable partnerships (new clinics, improved networking, better governance)
Building evidence base (data management services, CQI, e-health)
Better educated towards chronic diseases
Better health outcomes; improved life expectancy better quality of life / wellbeing
Better lifestyle choices: diet, exercise, smoking
Provision of appropriate care to manage chronic conditions
Reduced risk factors for disease & injury
Reduced complications / fewer hospitalisations / saving to health care system
Building cultually aware workforce
More effective / productive workforce
Building sustainable partnerships
Greenfield and brownfield clinics; improved governance
Building the evidence base
Data management services
Contribution to Indigenous health and social policy

34. Example: Institute of Urban Indigenous Health
The IUIH activities are diverse, multifaceted and lead to a range of outputs.
The impacts of these activities are closely aligned with the strategic goals of the IUIH: to improve access to comprehensive primary health care; to develop an effective and culturally aware workforce; to build sustainable partnerships; and to contribute to building the evidence base.
While some of these impacts may lead to quantifiable economic return, others are more difficult to quantify.
The next logical step for the Institute is to unpack the rich clinical data that it collects to demonstrate longitudinal improvements in patient and community outcomes from the range of Institute activities.
A better understanding of this clinical data would facilitate a more comprehensive assessment of the economic benefit of the IUIH and a better understanding of the IUIH contribution to closing the gap

35. Summary
In an environment of limited resources it is important to evaluate what we do
Economics provide a framework to identify value for money ranging from cost-effectiveness to cost-benefit analysis
Cost-effectiveness is appropriate when comparing health programs
Cost benefit is appropriate when examining return on investment or conducting an impact assessment
Policy makers are increasingly requiring evaluations that make sense – what is the return on the investment?
Good evaluation requires good data, plausible assumptions and a robust methodology

36. Thank you