1. The Appraisal, Extraction and Pooling of Cost and Cost Effectiveness Studies
JBI/CSRTP/ /0005
Welcome all participants to this module on Appraisal, extraction & pooling of Cost and cost effectiveness studies.

2. Introduction Recap of Introductory Module
Developing a question (PICO)
Inclusion Criteria
Search Strategy
Selecting Studies for Retrieval
This module considers how to appraise, extract and synthesize evidence from Cost and Cost Effectiveness studies.
Point 1/Top (reduced) section of schematic:
Provide summary of contents of previous day.
Introduction to JBI and Systematic review process and in particular those parts of the process which are ‘generic’ or ‘standard’ in any systematic review, independent of the evidence which is being sought and synthesised.
We examined the process of developing an appropriate question for systematic review, one of clinical relevance, using the PICO mnemonic/process. How the review question dictates the inclusion and exclusion criteria for the review and how it forms the basis for the keywords and terms used in the search strategy and which databases/sources the search will be directed towards.
The process of selecting studies for retrieval and roles/possibilities of your secondary reviewer were also discussed.
We also began the process of incorporating all of these activities/steps into your review protocol which your have begun to develop.
Point 2/Bottom section of schematic:
So it is at this first point, the critical appraisal of your retrieved studies, that the Modules in the CSR training program deviate. In this Module, we focus primarily on evidence from studies focused on establishing costeconomic evaluation studies, that is evidence which aims to establish cost-effectiveness, cost-utility or cost-benefit. The Cost-effectiveness (CEA) studies, cost-utility (CUA) studies, and cost-benefit (CBA) are the preferred design for this type of evidence.
To be able to critically appraise this type of evidence, you will need to understand about study design so will spend most of the morning and after lunch discussing aspects of study design inherent to the evidence or knowledge the primary research question is trying to establish.
After which we will move onto data extraction, or “pulling out” the information and values of interest for our review question and then we’ll finish the Module tomorrow with consideration of the synthesis of economic evaluation data.
Throughout the Module, you’ll have the opportunity to put what you learn into practice, critically appraising papers, extracting data, and then analysing and synthesising data using the JBI ACTUARI software. At the end of the day tomorrow, you’ll have some time to complete your protocols related to economic evidence, and everyone will have the opportunity to present their protocols to each other.

3. Aim and Objectives
The objectives of this module are to prepare participants to:
critically appraise studies of cost and cost effectiveness,
extract data from cost and cost effectiveness studies,
summarize the results of cost and cost effectiveness studies.
This module will take a step-by-step approach to reviews of Cost and Cost Effectiveness evidence. The objectives of this module are to prepare reviewers with a basic grounding in the methods and analytic techniques for cost and cost effectiveness studies. From this basis, the module moves in to the methods of systematic review starting with critical appraisal, then data extraction and synthesis of results.
It might be useful at this stage to check whether this is where participants were expecting to the module to head - most people will say yes, so don’t dwell on this for long, just get a brief confirmation of the general level of expectation.

4. Program Overview
Timetable for Day 1 – Appraising economic evaluation evidence and data extraction and we will do activities for each.

5. Session 1: Introduction to review of evidence on cost and cost effectiveness
This is the starting point of Session 1. In this session we will be focusing on the types of studies and methods of analysis common to cost and cost effectiveness studies, it includes examples of the types of questions associated with the different cost and cost effectiveness designs that are included in the module.
It should be noted that although costs and cost effectiveness are crucial to our understandings from a decision making perspective, there are limited study designs available to respond to questions in this field as compared with clinical effectiveness.

6. Common study designs
Prospective experimental or quasi experimental effectiveness studies with cost or cost effectiveness components
Modelling studies
Studies can be classified in a number of ways, one such approach is to consider which are comparative studies (either prospective or retrospective) and which are economic modelling studies – modelling studies may not be based on real patient data, but often draw on scenarios developed by the researcher.
In a prospective health economic evaluation a number of patients are receiving the intervention or the comparator and all clinical and economic data are collected and recorded for the patients in both groups, particularly as part of randomized trials.
Even when trial-based economic evaluation exist, modelling is likely to be undertaken to extend the analysis beyond the observed time periods to allow for the assessment of longer-term impacts. Decision analytical modelling compares the expected costs and consequences of decision options by synthesising information from multiple sources and applying mathematical techniques, usually with computer software. Different analytical methodologies such decision tree models, state-transition models, microsimulation models, types of dynamic models (such as difference equations models) can be found in literature. Markov models are special types of state-transition models. One important consideration is whether there is a need to explicitly model the individual patient or to consider the average experience of a cohort of patients. Models can be either deterministic models or stochastic (probabilistic) to reflect the uncertainty in the input parameters of the decision model. 6

7. Four approaches to analysis
Cost-minimization analysis (CMA);
Cost-effectiveness analysis (CEA);
Cost-utility analysis (CUA);
Cost-benefit analysis (CBA).
The types of analysis listed are the common, or dominant approaches within the health sciences and each of these will be discussed in detail on the following slides, including examples of each type of analysis. The methods of analysis sit within the designs from the previous slide, there can be a prospective CMA, or retrospective CMA, or a prospective or retrospective CUA, CEA, or CBA. 7

8. Methods, measures, benefits
Types of studies
Costs or measures
Benefits or Consequence measures
Comments
Cost Minimization Analysis (CMA)
Costs measured in monetary units (e.g.. Dollars)
Not measured
CMA is not a form of full economic analysis, the assumption is that benefits or consequences are the same, therefore the preferred option is the cheapest
Cost Effectiveness Analysis (CEA)
Benefits measured in natural units (e.g.. mmHg, cholesterol levels, symptom free days, years of life saved)
Results are expressed as dollars per case or per injury averted. Different incremental summary economic measures are reported (e.g.. Incremental cost-effectiveness ratio)
Cost Utility Analysis (CUA)
Benefits expressed in summary measures as combined quantity and quality measures (e.g.. QALY, DALY etc)
Two dimensions of effects measured (quality and length of life); results are expressed for example as cost per QALY
Cost Benefit Analysis (CBA)
Benefits measured in monetary units (e.g.. Dollars)
Benefits are difficult to measure monetarily, values used are Net Present Value (NPV) and Benefit Cost Ratio (BCR)
To date, these are the four main methodologies that have emerged...
Cost Minimization is considered a type of partial analysis as clinical outcomes from the interventions or programs being compared are assumed to be equal, only the costs are truly examined. Because the outcomes are considered to be the same, the dominant factor is a focus on which intervention (or program etc) is the cheapest.
Cost Effectiveness examines both the outcomes, and the costs associated with the interventions. In the table of methods, measures and benefits, you can see that Cost Effectiveness reports benefits as natural units (or clinical outcomes as the case may be). A limitation of Cost Effectiveness studies is that programs that achieve different outcomes can not be compared.
Cost-Utility measures in costs as a monetary benefit, while reporting on Quality of Life Adjusted Years (QALY) as a measure of both the quality and duration of life.
Cost-Benefit involves identifying the costs and benefits in monetary terms, then generating a summary measure that brings the cost and benefits together, commonly expressed as the NPV, or BCR.

9. Cost-minimization analysis (CMA)
In cost-minimization analysis (CMA) only the costs of the interventions are compared; the outcomes are assumed to be equivalent.
Cost-minimisation analysis (CMA) is not a form of full economic evaluation.
In cost-minimisation analysis the outcomes are assumed to be equivalent. In CMA only the costs of the interventions are compared. The preferred option is the cheapest. CMA cannot be used when outcomes of interventions are different.
The strength of each CMA lies in the acceptability that outcomes are indeed equivalent.
A common example of a cost-minimisation analysis is comparing two generic medications that are rated as equivalent. If the drugs are equivalent to each other only the differences in the cost are used to choose the one that provides the best value.
It would not be appropriate to compare different classes of medications using cost-minimisation analysis if there are noted differences in outcomes.
Another common example of a CMA is comparing the costs of receiving the same medication in different settings (for example hospital versus home; inpatient versus outpatient). 9

10. PICO Questions – Cost Minimization
What is the evidence on costs (direct and indirect) of laparoscopic compared to open appendectomy for patients aged 15 years or over (assuming the long-term outcome is the same in both groups)?
Being a CMA question: Ask the group for their comment and feedback to the following questions
what is the unit by which the outcome will be measured?
How will the benefits be measured?
What does this question tell you about the population of interest?
What does this question tell you about the primary outcome of interest?
Is the primary outcome of interest congruent with a cost minimization design?

11. Cost-effectiveness analysis
Costs are measured in monetary units;
The outcome is common to both alternatives but the effect size and direction may vary;
Outcomes are measured in natural/clinical units;
(e.g. mortality, myocardial infarctions, lung function, weight, bleeds).
Cost effectiveness analysis is an economic analysis that converts effects into health terms and describes the costs for some additional health gain (e.g. cost per additional stroke prevented). The effects of an intervention (and its comparators) are measured in identical units of outcome (e.g. mortality, myocardial infarctions, lung function, weight, bleeds, secondary infections). Alternative interventions are compared in terms of ‘cost per unit of effect’. 11

12. Cost–effectiveness Plane
A four-quadrant figure of cost difference plotted against effect difference:
quadrant I, intervention more effective and more costly than comparator;
quadrant II, intervention more effective and less costly than comparator;
quadrant III, intervention less effective and less costly than comparator; and
quadrant IV, intervention less effective and more costly than comparator. (Culyer, 2005:77-78)
The concept of cost-effectiveness is illustrated graphically on the CE plane. Strategies are represented according to their effectiveness (e) and cost (c) relative to those of some reference standard. A straight line with slope K passes through the origin, where K is the maximum acceptable cost-effectiveness ratio. Strategies plotted to the right of line K are cost-effective, while those plotted to the left are not. Furthermore, the most cost-effective of two or more mutually exclusive alternatives is that plotted the greatest distance to the right of K. The CE plane can be used to resolve ambiguities about the meaning of cost-effectiveness and to illustrate its relevance to decision making. Cost effectiveness studies tend to focus on individual benefits. 12

13. Cost Effectiveness PlaneQ4 Q1
In cost effectiveness analysis one is faced with the difficulty of weighting up the analysis of results of interventions from costs and effectiveness data. In some cases, it may be easy to choose where one intervention is superior in relation to cost and effect, however, in other cases, the most effective intervention may also be the most expensive. This plane facilitates understanding how to use the results to decide which intervention is the most effective.
The Cost effectiveness plane is represented by the dotted line that shows at which points the new intervention would be most likely to be acceptable, and where the decision might be less clear, higher cost and/or less effectiveness creates doubt that might require further study or investigation. Contrastingly, outcomes that fall below the dotted line suggest it is reasonable to accept the new intervention.
Consider the case of assessment of new drug B compared with existing drug A.
If drug B has a lower total cost (all related costs generated over the long term) and higher QALYs than drug A, it is easy to reach a conclusion - drug B is the obvious preference.
Conversely, there would be no reason to select drug B over drug A if the former had a higher total cost but delivered a lower QALYs.
The decisions become more difficult if drug B has a higher total cost but also affords a higher QALYs. In other words, if for example drug B is more effective (QALYs) than drug A, but also has a proportionately higher cost. Q3 Q2 13

14. PICO Questions – Cost Effectiveness
What is the cost effectiveness of percutaneous coronary intervention with drug-eluting stents (PES) compared to bare-metal stents (BMS) to reduce angina symptoms for patients undergoing single-vessel percutaneous coronary intervention?
Being a CEA question: Ask the group for their comment and feedback to the following questions
what is the unit by which the outcome will be measured?
How will the benefits be measured?
What does this question tell you about the population of interest?
What does this question tell you about the primary outcome of interest?
Is the primary outcome of interest congruent with a cost effectiveness design?

15. Cost-utility analysis
Costs are measured in monetary units;
Outcomes are common to both alternatives;
Effect size and direction may vary;
Outcomes are measured as healthy years (typically measured as quality-adjusted life-years (QALYs)).
Cost-utility analysis (CUA): when alternative interventions produce different levels of effect in terms of both quantity and quality of life (or different effects), the effects may be expressed in utilities. Utilities are measures which comprise both length of life and subjective levels of well-being. The best known utility measure is the quality-adjusted life year, or QALY. Alternative interventions are compared in terms of cost per unit of utility gained (e.g. cost per QALY). The benefit of this as compared with cost effectiveness is that the level of interest is not only on patients, but at a societal level. 15

16. Outcome measures for CUA
The primary outcome for CUA is expressed as quality-adjusted life-years (QALYs);
Other generic outcome measures for CUA:
Disability-adjusted life-year (DALY);
Healthy years equivalent (HYE);
Saved-young-life-equivalent.
(Drummond et al., 2005:14)
The main outcome measure reported in papers in qaly, some papers will report other outcomes.
Healthy years equivalent can be understood to mean: The number of years of perfect health (followed by death) that has the same utility as (is seen as equivalent to) the lifetime path of health states under consideration.
Disability adjusted life year can be understood to mean a measure of overall disease burden. Originally developed by the World Health Organization, it is becoming increasingly common in the field of public health and health impact assessment (HIA). It is designed to quantify the impact of premature death and disability on a population by combining them into a single, comparable measure. In so doing, mortality and morbidity are combined into a single, common metric.
Saved young life equivalent is less commonly applied, the principal with each approach is to aware of the terminology and its link to data that will be of use in your review. 16

17. PICO Questions – Cost Utility
What is the cost-utility of the cochlear implant in adults (age >18 years) with profound bilateral, post-lingual deafness compared with no intervention?
Being a CUA question: Ask the group for their comment and feedback to the following questions
what is the unit by which the outcome will be measured?
How will the benefits be measured?
What does this question tell you about the population of interest?
What does this question tell you about the primary outcome of interest?
Is the primary outcome of interest congruent with a cost utility design?

18. Cost-benefit analysis
Costs are measured in monetary units;
Outcomes are identified as single or multiple effects;
The effects are not necessarily common to both alternatives;
Outcomes are measured in monetary units.
In this approach, not only the cost, but also the benefits are expressed in monetary units, the problem with this approach is that heroic assumptions have to be made in the assumptions related to the translation of effects in to monetary units. For example in policy, choices must be made with regard to the monetary values associated with the outcomes of providing dental services for older adults, versus the monetary outcomes of providing a new pediatric cancer centre.
Cost benefit only provides a monetary value, other attributes are not considered, leading to quite subjective comparisons informed by a monetary value that may not reflect societal values. 18

19. PICO Questions – Cost Benefit
What is the cost-benefit of donepezil compared to galantamine for cognitive function in patients with mild to moderate Alzheimer’s disease?
Being a CBA question: Ask the group for their comment and feedback to the following questions
what is the unit by which the outcome will be measured?
How will the benefits be measured?
What does this question tell you about the population of interest?
What does this question tell you about the primary outcome of interest?
Is the primary outcome of interest congruent with a cost benefit design?

20. Searching for Evidence
Cost and Cost Effectiveness keywords
Clinical keywords
General databases
Specific databases
Because these reviews cover costs and cost effectiveness within the one review, you may need to search for these arms of the review in separate sessions, then bring your results together. Although the ideal is to find studies that cover both aspects of the intervention of interest.
Searching for economic evidence should use the same structures as taught in the Introductory module. The structure will rely on methodological descriptions that are more specific to economics, and these are described in detail in the review handbook
The difference in structure is that there are economic as well as clinical words to incorporate in your search matrix. This means search strings will include a series of economic methodological terms and free text words, making the strings longer – it pays to map them carefully to ensure no lines are missed, and that Boolean terms are used correctly.
As you can see from this slide, the core processes are the same as for any type of evidence. There are though some differences, and these relate to the volume of results likely to be obtained from different sources

21. Specific economic databases
NHS Economic Evaluation Database (NHS EED)
Health Economic Evaluation Database (HEED)
Cost-effectiveness Analysis (CEA) Registry
Health Technology Assessment (HTA) database
Paediatric Economic Database Evaluation (PEDE)
European Network of Health Economic Evaluation Databases (EURONHEED)
COnnaissance et Decision en Economie de la Sante (CODECS)
Economic evidence is represented across most of the general databases, however, there is not necessarily a strong or comprehensive representation of economic studies in any particular main stream database.
It is therefore useful to search databases specific to the methodology. These are some of the databases you would take note of if undertaking a review of economic evidence. There may be more that you are aware of, and one additional approach is also to talk with experts, to gather opinions on whether there are particular databases that should be used.

22. Group Work 1: Identification of Economic Evaluation Study Designs
Refer to Workbook.
Report back
Group Work 1 from Workbook pages 3-8.
Report back 22

23. Session 2: Critical Appraisal of Cost and Cost Effectiveness Studies
We will continue our discussion on critical appraisal having spent the first session considering types of evidence and structures for economic questions. Those attributes of economic reviews then lead us through the search process and in to how quality, or internal validity is considered in economic literature.

24. Why Critically Appraise?
1004 references
832 references
Scanned Ti/Ab
172 duplicates
117 studies
retrieved
715 do not meet
Incl. criteria
82 do not meet
35 studies for
Critical Appraisal
Why Critically Appraise?
Combining results of poor quality research may lead to misleading understandings of issues explored
Thinking back to Module 1 of this program, when we went through the process of study selection, our results showed that we had 35 studies that met the inclusion criteria and made their way onto appraisal.
Why? Simply, there is an overwhelming amount of scientific literature available, however, not all of this literature is of high quality. All of the papers you ultimately select for inclusion into your review must then go through a rigorous appraisal process by 2 members of your review team.
The aim is to include only studies which are of a high standard and exclude those which are of poor quality.
Inclusion of poor quality research may lead to biased or misleading conclusions in your review! 24

25. The Critical Appraisal Process
Every review must set out to use an explicit appraisal process. Essentially,
A good understanding of research design is required in appraisers; and
The use of an agreed checklist is usual.
To be able to use a clear, well defined appraisal process, an appraiser must have a good understanding of research design. Without a good understanding, it will be difficult for an appraiser to answer the questions posed by Appraisal checklists accurately and confidently.
There are many agreed and accredited, tried and tested, checklist available - these are the tools that should be used. Such as the JBI checklists.
Practically speaking, what we strongly recommend is that BEFORE 2 reviewers commence independent critical appraisal of papers, that they work together to read the list of appraisal items and ensure that their understanding of what they mean, and how they will be interpreted/applied is the same or at least similar. This training program in and of itself is not enough, practical conversations are a necessary part of the process. 25

26. Critical appraisal of cost and cost effectiveness evidence
Primary purpose of critical appraisal is to assess a study’s quality and determine the extent to which a study has excluded the possibility of systematic flaws in its design, conduct and analysis.
The main object of critical appraisal is to assess a study‟s internal validity and determine the extent to which a study has excluded the possibility of systematic flaws in its design, conduct and analysis. If a study has not excluded the possibility of flaws, then its results are questionable and could well be invalid.
Therefore, part of the systematic review process is to evaluate how well the potential for flaws has been managed within a study. The results of this process give us an indication of which papers are considered more robust, or as having greater internal validity - with the eventual aim of only including high quality studies in the resulting systematic review.
A secondary although no less strategic benefit of critical appraisal is to take the opportunity to ensure each retrieved study has included the population, intervention and outcomes of interest specified in the review.
There are a variety of checklists and tools available to assess the validity of studies. Most of these use a series of criteria that can be scored as being met, not met or unclear. The decision as to whether or not to include a study can be made based on meeting a pre-determined proportion of all criteria, or on certain criteria being met. It is also possible to weight the different criteria differently. These decisions about the scoring system and the cut-off for inclusion should be made in advance, and be agreed upon by all participating reviewers before critical appraisal commences.
The assessment tools included in the JBI software analytical module ACTUARI are required for all JBI entities conducting economic reviews through JBI. 26

27. Is there a well defined question?
JBI Critical Appraisal Checklist for cost and cost effectiveness studies
Is there a well defined question?
Is there a comprehensive description of alternatives?
Are all important and relevant costs and outcomes for each alternative identified?
Has clinical effectiveness been established?
Are costs and outcomes measured accurately?
Are costs and outcomes valued credibly?
The following slides provide a brief overview of JBI critical appraisal approach for economic evaluation studies. You will present the details in the following slides, so only give a brief run through on these 2 slides.
Keep in mind that the reviewer may need to consider whether each question is relevant or applicable to their question, eg, 3. would not be applicable in cost minimisation studies.
From this slide, you can see that the initial questions deal with elements of the costs, but also with the clinical effectiveness of interventions. The remainder of the questions on the following slide are focused on the cost perspective. However, the cost perspective is based on the clinical, therefore appraisal of both is important. 27

28. JBI Critical Appraisal Checklist for cost and cost effectiveness studies
Are costs and outcomes adjusted for differential timing?
Is there an incremental analysis of costs and consequences?
Were sensitivity analyses conducted to investigate uncertainty in estimates of cost or consequences?
Do study results include all issues of concern to users?
Are the results generalizable to the setting of interest in the review?
Now that you have had this brief overview of the structure of the JBI appraisal instrument, we will take a more structured look at each of the 11 questions. 28

29. JBI Economic evidence appraisal
Is there a well defined question?
Costs and effects;
Comparison of alternatives;
Perspective of the analysis (including the decision-making context).
The well defined question may include the attributes on screen, but this will vary with the specific study design being assessed.
A well defined question will be congruent with the type of economic study, whether that be a cost minimisation, utility, effectiveness or benefit study. We now know that there are different methods of costs as well as of benefits depending on study type. Cross reference the study type with the summary table on methods, measures and benefits from Session 1 . Does the structure of the question indicate that both costs and effects were studied, or just costs? Was a comparator described.
The perspective relates to the generalizability of the findings. If the perspective is limited for example to just a health sector or system, costs and benefits from other systems, such as the education system, or social welfare systems are excluded from consideration. If the perspective is at the population or societal level, then costs and benefits across systems are considered, and the perspective is wider and hence more applicable across systems and costs.
Questions that will assist you in addressing this criterion (Drummond, 1990):
Did the study examine both costs and effects of the services or programs?
Did the study involve a comparison of alternatives?
Was a viewpoint for the analysis stated or was the study placed in a particular decision-making context?

30. Effects Mortality measurements; Morbidity measurements;
Health-related quality of life measurements.
There are different types of outcomes/benefits reported in economic evaluation studies expressed in terms of mortality measurements (survival, survival at 1 year, survival at 5 years), morbidity measurements (cure of the disease, eradication of the infection, stroke avoided, fractures avoided, complications avoided) and health-related quality of life measurements (QALYs).
In cost-minimisation analysis the outcomes are assumed to be equivalent. In CMA only the costs of the interventions are compared. Cost-effectiveness analysis (CEA) measures outcomes in natural units: mmHg, cholesterol levels, symptom-free days, years of life saved. In Cost-utility analysis (CUA) there are two effects measured (quality and length of life) whose product is taken as quality-adjusted life years (QALYs). In cost-benefit analysis (CBA) not only are costs valued in monetary terms so are the benefits.

31. Perspective
The ‘viewpoint’ adopted for the purposes of an economic appraisal (cost–effectiveness, cost–utility studies and so on) which defines the scope and character of the costs and benefits to be examined.
When reading papers you will be extracting data or taking notes on the perspective that is being addressed, some papers will seek to cover a range of perspectives to be comprehensive in their analysis. While this is helpful, papers that do not cover multiple perspectives should not be penalisled or excluded unnecessarily. Just remain aware that the extent of the perspective has implications for how the costs and benefits can be applied, the narrower the perspective (ie a focus on a health service) the less applicable the results will be, and the more likely that costs not reflected in that health service may be missed. A wider perspective eg a population level perspective will cover not just the health service costs but also wider societal costs and benefits, and will therefore be more applicable

32. Perspectives Societal perspective; Health sector perspective;
Other sector perspective;
Health insurance perspective;
Hospital perspective;
Patient perspective.
The perspective is the economic term that describes whose costs are relevant based on the purpose of the economic evaluation study. It is necessary to state the perspective of the study as this determines which costs are included.
The ideal perspective is considered to be the societal perspective. This is the widest perspective because it looks at the costs from the viewpoint of society as a whole. The societal perspective includes direct, indirect and intangible costs.
There are different perspectives that can be applied in economic evaluation studies:
 Societal perspective (the impact of an intervention on the welfare of the whole of society, not just the individuals or organizations directly involved)
 Health sector perspective (costs and benefits limited to the health sector only);
 Other sector perspective (may be government, NGO, any agency where costs and benefits are being investigated) ;
 Health insurance organization/institution perspective;
 Hospital perspective;
 Primary care sector perspective;
 Social services perspective (a branch of social welfare, not the same as a societal perspective);
 Patient perspective.
Costs will increase as the perspective widens. Using narrower perspectives allows cost shifting to take place from secondary care to primary care or from the healthcare provider to other public sectors. This may save costs in one area but increase costs or burden in another area.

33. JBI Economic evidence appraisal
Is there a comprehensive description of alternatives?
Important alternatives
Do-nothing alternative
It is important when measuring costs to have a good understanding of the procedure or intervention. Hence in critical appraisal it is important to establish if the description of alternatives is reasonable, or adequate to inform understandings of the characteristics of the comparators. To arrive at an accurate outcomes of costs, you need a detailed understanding of the intervention and its comparator.

34. JBI Economic Evidence Appraisal
Are all important and relevant costs and outcomes for each alternative identified?
Was the range wide enough for the research question;
Does it cover all relevant perspectives;
Were capital as well as operating costs included.
Lets first consider the comprehensiveness of costs. To be able to answer this question, it is important to understand the range of costs that health economists consider. These are expanded upon in the following slides.
Importance and relevance are the key concepts in question 3 of the appraisal instrument. Identifying the types of costs (capital, recurrent, fixed, variable, semi-fixed) will help to answer this question. Also, knowing the “perspective” will help inform the types of costs and outcomes that should be considered relevant.
Capital costs include the cost of equipment, vehicles, buildings and one-time training programmes.
Recurrent costs are the value of recurrent resources. Recurrent resources are those with useful lives of less than one year and have to be purchased at least once a year – yearly, monthly, weekly, daily or irregularly but frequently.

35. Typical classification of costs
Functional costs;
Financial and economic costs;
Direct, indirect and intangible costs;
Capital and recurrent costs;
Fixed and variable costs;
Opportunity costs
How the costs are classified in an economic study sets the basis for the analysis of a program. The classifications listed here are indicative of how costs might be reported in papers. The next few slides will look at each of these in more detail.
Opportunity costs are not expressly covered in this module.

36. Functional costs Can be classified into categories: personnel;
buildings and space;
equipment;
supplies and pharmaceuticals;
transportation;
training;
information, education and communication.
Functional Costs costs are associated with a specific activity, the list on the PowerPoint reflects some of the functional activities undertaken in health care settings. Functional costs are useful in that they allow services to determine prices after all functional costs have been allocated

37. Financial and economic costs
Financial costs are defined as the actual money spent on the resources;
Inclusion of the costs of all resources, regardless of their financial cost is known as the economic cost.
The temptation when reading papers is to focus on actual costs - real dollar transactions, and you read in the media at fairly regular intervals about the cost of a piece of equipment usually the same article has someone claiming how much time they will save, or how much more service they will deliver.
The Financial cost is the cost of the equipment, however the economic cost is wider - it includes the actual monetary cost, but also includes the less tangible costs such as time, service delivery costs etc - all resources have costs associated with them that extend beyond a direct dollar purchase figure. When reading papers be aware of this and look for wider costs.

38. Direct, indirect and intangible Costs
Direct costs are associated directly with a healthcare intervention (e.g. drugs, staffing);
Indirect costs refer to the productivity gains or losses (e.g. time off work, illness);
Intangible costs refer to the non-monetary assets that can not be readily seen (e.g. anxiety, fatigue, pain or suffering from an illness or treatment).
The correct identification, measurement and valuation of costs is essential in health economics.
Economic studies use a range of costs. It is important to be able to distinguish between the different types of costs that are used. There are two basic approaches for cost categorization:
 Cost categorization into direct medical costs, direct nonmedical costs, indirect costs, and intangible costs;
 Cost categorization into health care sector costs, other sector costs, patient and family costs, and productivity costs.
Direct costs are those costs associated directly with a healthcare intervention. Direct costs represent the value of all goods, services, and other resources that are consumed in the provision of an intervention or in dealing with the side effects or other current and future consequences linked to it.
Direct medical costs represent the value of health care resources (e.g., tests, drugs, supplies, health care personnel, and medical facilities) consumed in the provision of an intervention or in dealing with the side effects or other current and future consequences linked to it. Direct nonmedical costs represent the value of nonmedical goods, services, and other resources, such as child care and transportation, consumed in the provision of an intervention or in dealing with the side effects or other current and future consequences linked to it
Indirect costs refer in economics to the productivity gains or losses related to illness or death. In accounting, indirect costs is a term used to describe overhead or fixed costs of production. Indirect costs are incurred by the reduced productivity of a patient and their family resulting from illness, death or treatment. Indirect costs may include time off work or housekeeping, time spent going to healthcare providers, time spent caring for the patient by relatives or paid carers, time forgone from leisure, early retirement.
Examples of indirect costs are: change in productivity, lost productivity while on the job, income lost by family members, foregone leisure time, time spent by patient seeking medical services, time spent by family and friends attending patient, psychosocial costs, valuations others put on patient’s health and wellbeing, pain, changes in social functioning and activities of daily living.
Intangible costs include anxiety, fatigue, pain or suffering from an illness or treatment.
Health care sector costs include medical resources consumed by health care entities. These types of costs are similar to the definition of direct medical costs but do not include direct medical costs paid for by the patient or other non-health care entities.
Other sector costs are the costs associated with the impact of the disease and the treatment on other sectors such as housing, homemaker services, educational services, public assistance, prison system.
Patient and family costs include the patient’s or family’s share of direct medical as well as direct nonmedical costs.
Productivity costs are the costs associated with lost or impaired ability to work or to engage in leisure activities due to morbidity and lost economic productivity due to death.

39. JBI Economic Evidence Appraisal
Has clinical effectiveness been established?
Was this through experimental research?
If so did the trial protocol reflect what would happen in regular practice?
Was effectiveness established through a synthesis of clinical studies?
Were observational data or assumptions used to establish effectiveness?
If so what were the potential biases in results?
Was there evidence that the program’s effectiveness has been established? Was this done through a randomized, controlled clinical trial, or a systematic review? If not, what type of evidence was used and how valid and reliable (as well as precise and strong) was the evidence of effectiveness? The effectiveness component may be within the same publication, in a companion publication, or alternate publication. The criteria related to evidence of effects is not relevant to cost minimization studies.

40. JBI Economic Evidence Appraisal
Are costs and outcomes measured accurately?
Were any of the identified items omitted from the measurement?
If so does this mean that they carried no weight in the subsequent analysis?
Were there any special circumstances (e.g. joint use of resources) that made measurement difficult?
If so, were these circumstances handled appropriately
The kinds of considerations you should discuss with your co-reviewer related to question five relate to which costs and benefits were included or excluded, and whether those included were measured reliably.
Were costs and consequences measured accurately in appropriate physical units (e.g., hours of nursing time, number of physician visits, days lost from work, years of life gained) prior to valuation?
Were any identified items omitted from measurement? If so, does this mean that they carried no weight in the subsequent analysis?
Were there any special circumstances (e.g., joint use of resources) that made measurement difficult? Were these circumstances handled appropriately?

41. JBI Economic Evidence Appraisal
Are costs and outcomes valued credibly?
Were the sources of all values clearly identified?
Possible sources include market values, patient or client preferences and views, policy maker’s views and health professional’s judgements;
Were market values employed for changes involving resources gained or depleted?
Where market values were absent (e.g. volunteer labour) or did not reflect actual values (such as clinic space donated at a reduced rate) were adjustments made to approximate market values?
Because it is so difficult to arrive at accurate measures of economic costs and outcomes, and hence a range of assumptions will be made. In critical appraisal it is important to ask whether there has been an attempt to value costs and outcomes in the most transparent and accurate way (as is appropriate to the study design).
These sub points provide some guidance as to the types of issues that it would be useful to consider, its not a comprehensive list, and you may be aware of alternate points. When undertaking appraisal, it is worthwhile to discuss these questions with your co-reviewer and ensure that you have a common understanding of how these assumptions will be interpreted.

42. JBI Economic Evidence Appraisal
Are costs and outcomes adjusted for differential timing?
Were costs and outcomes that occur in the future ‘discounted’ to their present values?
Was there any justification given for the discount rate used?
Were costs and consequences that occurred in the future discounted to their present values?
Was any justification given for the discount rate used? The rates used, and approaches taken may vary, but some form of discounting should be reported in the paper
Discounting is a procedure for reducing costs or benefits occurring at different dates to a common measure by use of an appropriate discount rate. Discounting makes current costs and benefits worth more than those occurring in the future because there is an opportunity cost to spending money now and there is desire for benefits now rather than in the future.
To take into account the opportunity cost of investing now rather than waiting one year we have to discount future costs. Therefore, if two healthcare interventions both released £100 in savings but for one we had to wait a year, then, all other things being equal, we would adopt the intervention that saved £100 now.

43. JBI Economic Evidence Appraisal
Is there an incremental analysis of costs and consequences?
Were the additional (incremental) costs generated by one alternative over another compared to the additional effects, benefits or utilities generated?
This question asks whether additional (incremental) costs generated by one alternative were compared to the additional benefits, costs or utilities generated in comparison with another alternative. The incremental analysis is therefore, the analysis of the difference between the two alternatives. It is useful for decision makers to have information about how costs and effects change as the context (variables/inputs) change. Often in modelling studies, the results of differing scenarios are presented, and hence it is important to ask this question, particularly of studies that use modelling. 43

44. JBI Economic Evidence Appraisal
Were sensitivity analyses conducted to investigate uncertainty in estimates of costs or outcomes?
If a sensitivity analysis was employed, was justification provided for the range of values (or for key study parameters)?
Were the study results sensitive to changes in the values (within the assumed range for sensitivity analysis or within the confidence interval around the ratio of costs to outcomes)?
Uncertainty in economic studies may arise from imprecise estimates of cost or charges. A sensitivity analysis examines a range of estimates for the key variables and assesses impact on the study results. Review each paper to identify whether sensitivity analysis was conducted and examine not just the significance, but the confidence intervals associated with the values.
In modelling studies the results are very dependent upon assumptions made about technology/productivity and prices for example. Sensitivity analysis tests the robustness of results to changes in the assumptions made. Papers that include this feature are considered superior.

45. JBI Economic Evidence Appraisal
Do study results include all issues of concern to users?
Are the results of cost and effect for the alternative interventions?
Do they clearly specify the relative size of the effects for the interventions?
Do they clearly show how costs differ for the two interventions?
Can we use them with the Cost Effectiveness Plane?
Is the study useful to users? Are the results based upon the comprehensive application of study methods for the specific design? Are the costs relative to the effects, is there sufficient coverage of costs and effects to inform the Cost and Cost Effectiveness Plane.

46. Did the study take account of other important factors in the choice or decision under consideration (e.g. distribution of costs or outcomes or relevant ethical issues)?
Did the study discuss issues of implementation such as the feasibility of adopting the preferred program given existing financial or other constraints and whether any freed resources could be re-deployed to other worthwhile programs? 46

47. JBI Economic Evidence Appraisal
Are the results generalizable to the setting of interest in the review?
Did the study make clear that the findings on costs and effects were generated in a specific setting using particular assumptions?
Was the generalizability of the results to other settings and patients/client groups discussed?
Discuss the factors limiting the transferability of economic data.
Factors limiting the transferability of economic data are:
demographic factors;
epidemiology of the disease;
availability of health care resources;
variations in clinical practice;
incentives to health care professionals;
incentives to institutions;
relative prices;
relative costs;
population values.

48. Group Work 2: Critical Appraisal of evidence from economic evaluation studies
Workbook
Report back
Group Work 2 from Workbook pages 9-11.
Report back

49. Session 3: Study data and Data Extraction
We have now discussed how to appraise the quality of economic evaluation studies. Once you have completed the appraisal process of your retrieved articles with your secondary reviewer it is time to turn your attention to extracting the relevant data from your included literature.
Data extraction does not refer solely to collecting the relevant numbers related to your outcomes of interest, although these are extremely important values for your review, but it also refers to the relevant descriptive data and the like which will be necessary to present in your review. All of this data can be extracted at the same time with use of the appropriate forms.

50. Data most frequently extracted
1004 references
832 references
Scanned Ti/Ab
172 duplicates
117 studies
retrieved
715 do not meet
Incl. criteria
82 do not meet
35 studies for
Critical Appraisal
26 studies incl.
in review
Data most frequently extracted
After critical appraisal in this example - 26 studies have been included and will appear in the body of the review itself - with specific details related to the outcomes of interest in the results section of the review. These details will also appear alongside each study citation In the table of included studies.
The most important data to extract however, the data which will have the most direct bearing on the review question are the ‘findings and illustrations of these findings you extract from the literature… 50

51. Considerations in Data Extraction
Source - citation and contact details
Methods - study design, concerns about flaws
Participants –number, characteristics and suitability for inclusion
Interventions - description of interests
Outcomes - outcomes and time points
Results - for each outcome of interest
Miscellaneous - funding source, etc
Data extraction is a multi phase process, one tends to begin with the general study characteristics and move to the particular data of interest in a later process. The ACTUARI software provides the framework for data extraction, follows this process and reflects these criteria.

52. ACTUARI: Data Extraction
Interventions and Comparator
Setting
Geographical context
Participants
Source of effectiveness data
Author’s conclusion
Reviewer’s comments
Clinical effectiveness results
Economic results
Interventions and Comparator
The „Interventions‟ field relates to the new treatment (or intervention) whose cost or effectiveness is being compared to the standard (or control, or „Comparator‟ treatment). The first time that you add information to either the Interventions or Comparator fields in JBI-ACTUARI, the scroll down menus will be blank. However, once you have entered some text into these fields, these options will be available in the scroll down menu of any future data extraction from other studies.
Setting
The Setting field relates to the location where the study took place. For example, was the study in a hospital, community or residential aged care setting?
Geographical context
The Geographical field relates to the region (city, state, country) in which the study took place. 31
Participants
The Participants fi eld should list the age, gender, race or cultural groupings of the people included in the study. You should also include the basis of why people were excluded from the study.
Source of effectiveness data
There are four options available to select from the scroll down menu in this field. They refer to the original location of the information from which the effectiveness of the intervention compared to the comparator was derived:
 Single Study (same participants);
 Single Study (different participants);
 Multiple Studies (meta-analysis);
 Multiple Studies (no meta-analysis).
Selection of a particular type of source document determines which data extraction fields become available in the next phase of extraction. This is described in detail below.
Author’s conclusion
Summarise the main findings of the study from the author‟s perspective.
Reviewer’s comments
Summarise your interpretation of the study and its significance. Once this data has been extracted and entered, the analytical module JBI-ACTUARI takes users to a second data extraction page specific to the methods described under “Source of effectiveness data”. There are two primary sections in this last step in data extraction. The first relates to the clinical effectiveness component of the study, the second to the data on economic effectiveness.
Clinical effectiveness results
This section relates to evidence on the clinical effectiveness of the intervention versus the comparator, or control, group. The fields in this section are designed for numbers and free text relating to the study design (eg randomized controlled study, cohort study, case control), the study date (in years), the sample size (in numbers, combining both treatment and comparator groups if relevant), the type of analysis used (eg intention to treat analysis), and the results of the clinical outcome. Note that both the study date and the sample size fields require numbers; non numerical data will not be saved to the ACTUARI database.
Economic effectiveness results
There are several fields in the economic effectiveness results section. The first relates to the date (year) when the economic data were collected. Note that only numerical data can be entered here. For multiple studies option, the next field relates to economic modeling used. The third field requires a list of the measurements of benefits that were used in the economic evaluation. The fourth, fifth and sixth fields relate to costs examined in the study: direct costs of the intervention/program being evaluated, indirect costs and the currency used to measure the costs. Another field requires information on statistical analysis of costs data. Another field relates to the results of any sensitivity analysis conducted as part of the study. One field relates to listing the estimated benefits to using the intervention instead of the comparator. Another field requires a summary of the cost results findings, and a final field is a summary of the synthesis of the costs and results.

53. First level extraction
Describe the fields, and that the drop down menus are actioned for all studies once they have been utilised in one study. Reviewers should avoid duplicating entries that will for the drop down menu items. 53

54. ACTUARI: Extracting data from economic studies
ACTUARI data extraction
Four options available for economic evaluation methods
Data extraction follows critical appraisal of papers that the primary and secondary reviewers have agreed inclusion on. Elements of data extraction for all studies are similar as the aim is to identify and extract data specific to a particular outcome of interest. The data extraction begins with recording of the method, identifying the setting and describing the characteristics of the participants. When data extraction of study background detail is complete, the extraction becomes highly specific to the nature of the data of interest and the question being asked in the review.
In JBI-SUMARI, elements of data extraction are undertaken through the analytical modules and the data extracted is automatically uploaded to JBI-CReMS. For economic reviews, synthesis is conducted in the analytical module, and the final report generating in JBI-CReMS.
The JBI ACTUARI extraction details page lists a range of fields which describe the study: economic evaluation method, interventions, comparator, setting, geographical context, participants, source of effectiveness data, author‟s conclusion, reviewer‟s comments and a field for whether the extraction details are „complete‟.
There are four options available in the „Economic Evaluation Method‟ field scroll down menu. The four options are:
 Cost minimisation – analysis or design that finds the least costly intervention/program among those shown or assumed to be of equal benefit.
 Cost effectiveness – analysis or design that compares an intervention or program having the same health outcome in a situation where, for a given level of resources, the desired outcome is the maximum health benefit to the population of interest.
 Cost utility – analysis or design in which interventions/programs that are targeted at different outcomes (in terms of quantity and quality of life) are compared, usually using units of quality-adjusted life-years or QALYs.
 Cost benefit – analysis or design in which costs and benefits are measured using monetary units and computes a net monetary gain/loss or a cost-benefit ratio.

55. Second level extraction
Although its difficult to see the fields in this slide, you can see that there are separate sections for the clinical effectiveness and the economic effectiveness components of papers. This is where the reviewer will drill down and extract the detailed information about these elements of the paper. 55

56. The outcome category is included in the detailed extraction, but is not actually an extraction of data. This is where you as a reviewer will, on the basis of your knowledge of a paper give an indication of where it sits in terms of costs and clinical effectiveness. You can come back to this screen and edit/update your decision at a later date.
Outcome category
The outcome category section is a three by three matrix of all possible outcomes of an economic evaluation. The ultimate decision about the clinical and cost effectiveness of the intervention under examination is entered here, using the data extracted above on both the clinical effectiveness and cost of the intervention.
In comparing the clinical effectiveness of two alternatives there are three possibilities: (i) the intervention of interest is better or more effective (ie a ‘+’) than the comparator, (ii) the intervention is equally effective (ie a ‘0’) or (iii) the intervention is less effective (ie a ‘-’). Similarly, in terms of cost, there are three possibilities: (i) the intervention is more expensive (ie a ‘+’), (ii) the intervention and comparator’s costs are the same (ie a ‘0’), or (iii) the intervention is less expensive (ie a ‘-’). Note that each of the comparisons between intervention and comparator can only be classed as one of nine options (A – I). For example, an intervention that was shown to be more effective and less expensive would be scored as ‘G’, whereas an intervention that was less effective and of equal cost would be scored as ‘F’.

57. Group Work 3 Data Extraction from economic evaluation studies;
Refer to Workbook;
Report back.
Group Work 3 from Workbook pages
Report back

58. Session 4: Protocol Development in CReMS
Timetable for Day 2 - Focus will be on data extraction and synthesis, and on how to use of the CREMS - ACTUARI software and on completing and presenting your protocols. Start the day with some open discussion where participants give their views and are able to raise questions for discussion.

59. Group Work 4 Develop a draft protocol in CReMS Refer to Workbook;
Group Work 4 from Workbook pages
Report back

60. Session 5: Synthesis/Reporting cost and cost effectiveness evidence
This session is about synthesis/reporting economic evidence. It follows the progression of the stages involved in a systematic review.

61. Synthesis/Reporting economic evidence
Presentation of results of synthesis:
Tables of results;
Narrative summary;
Hierarchical decision matrix.
Essentially, there are three options for the synthesis of economic data: results are presented in a narrative summary; results are presented in tables of results; or results can be summarised using a hierarchical decision matrix.
Synthesis should begin by constructing a clear descriptive summary of the included studies. This is usually done by tabulating details about study type, interventions, numbers of participants, a summary of participant characteristics, outcomes and outcome measures. An indication of study quality may also be given in tables.

62. Tabular summary of economic evidence
Examples of Tables of Results.

63. Narrative summary of economic evidence
“...The median and mean willingness to pay for a 25% reduction in symptoms were $US27 and $US87 per month (1997 values), respectively. Median and mean estimates nearly tripled for a 50% reduction. ...Willingness to pay of patients with urinary symptoms was between £74 and £92 per year (1999/2000 values) for complete continence with no adverse effects, substantially lower than in the Swedish[58] and US[40] studies. Individuals without symptoms valued this outcome at only between £14 and £21 per year.”
Examples of narrative summary of economic evidence.
All systematic reviews should contain text and tables to provide an initial descriptive summary and explanation of the characteristics and findings of the included studies. However simply describing the studies is not sufficient for a synthesis. The defining characteristic of narrative synthesis is the adoption of a textual approach that provides an analysis of the relationships within and between studies and an overall assessment of the robustness of the evidence.
The idea of narrative synthesis within a systematic review should not be confused with broader terms like narrative review, which are sometimes used to describe reviews that are not systematic.

64. ACTUARI decision matrix summary of economic evidence
From the data extraction, particularly the outcome specific data per included paper, reviewers are able to generate a matrix, which lists the comparison of interest, the score from the three by three matrix for each study (‘the dominance rating’) and the study citation. Discuss the matrix.

65. Session 6: Appraisal, Extraction and Synthesis using JBI-ACTUARI
Give a current progress update. This session is a practical, visual walk through of the software JBI and the Collaboration utilise for systematic reviews of costs and cost effectiveness.

66. Analysis of Cost, Technology and Utilization Assessment and Review Instrument (ACTUARI)
The program that JBI has developed as part of the SUMARI suite of software for economic evaluations is ACTUARI. It can be used in conjunction with CReMS as was illustrated in module one to develop the protocol, manage studies, allocate reviewers and draft the report.
The Joanna Briggs Institute Analysis of Cost, Technology and Utilization Assessment and Review Instrument (JBI-ACTUARI) has been designed to enable reviewers to systematically review and combine the results of economic evaluation healthcare primary research. The JBI-ACTUARI software facilitates the management, appraisal, extraction and analysis of the results of studies that investigate the economic evaluation of two or more alternatives of healthcare intervention or program. It is integrated into the Joanna Briggs Institute Comprehensive Review Management System software (JBI-CReMS) as a web-based database and incorporates a critical appraisal instrument, data extraction forms, the synthesis of similar studies function and a reporting function.

67. Each review has a ‘Primary’ and ‘Secondary’ reviewer
Each review has a ‘Primary’ and ‘Secondary’ reviewer. A Primary reviewer leads the review and has rights to add, edit or delete reviews; a Secondary reviewer assesses every paper selected for critical appraisal, and assists the primary reviewer in conducting the review.
The Reviews page lists all reviews that are in the system. This page is used to add new, edit existing reviews or delete reviews. Information about a review, including its title, a description of the review, the year the review was commenced, and the usernames of Primary and Secondary reviewers are all presented. An “Actions” column has been introduced which replaces the “Edit” column that was in the SUMARI v.4.0 modules. The status column of the review that was in the SUMARI v.4.0 has been removed from the "Reviews" page.
Only a reviewer can view the information on this page, and only reviews that correspond to the selected ‘Reviews radio buttons’ will be displayed. In the example only reviews in which the user is the Primary Reviewer and which are currently being conducted (‘open’) will be displayed. To access reviews as the Secondary Reviewer, click on the <Secondary> radio button on the left of screen. To access completed reviews, click on the <Closed> radio button.

68. Edit Reviews
From the main “Reviews” screen click on <Edit> in the “Actions” column alongside the appropriate review to modify review details. From here, you can edit the Review Title, Review Question, and Year. To save any changes, click the <Update> button. To exit without saving changes, selecting <Cancel> will take you back to the reviews page. Pressing the <Close> button will change the status of the review to closed. Click <Undo> to remove any text just added to any field on the screen.

69. Adding a new review via JBI-CReMS
All studies included in a review must be ‘assigned’ to one of the four analytical modules of the SUMARI software (JBI-QARI, JBI-NOTARI, JBI-ACTUARI, and JBI-MAStARI) via the Studies page in JBI-CReMS. Assigning a study to an analytical module will facilitate the assessment, appraisal and extraction of the study in that analytical module. A new review – viewable in JBI-ACTUARI – will be automatically created when a JBI-CReMS review has at least one of the studies included in the review assigned to JBI-ACTUARI.
When a “new Review” is entered into the system, whether via the ACTUARI analytical module or via CReMS, it will appear as the first record in the list of reviews, rather than the last as was the case across the SUMARI v.4.0 analytical modules.

70. Opening an existing review
To open an existing review, click on the title link under the ‘Review Title’ column in the review page (eg ‘Cost-Effectiveness of inhaled corticosteroids in asthma: a systematic review). By opening the review in this manner you are automatically selecting it. This will take you to the studies page.

71. Editing a study
Click on <Edit> to modify study details. This will bring you through to the same screen accessible via selecting the study name hyperlink and then clicking <Details> from the left hand box menu. You may edit citation details from this screen.

72. How to start critical appraisal
From the studies page, click on the authors’ hyperlink of a study which has ‘New’ in the Status column and ‘None’ in the Assessment column. This will take you to the assessment page.
If you are the Primary Reviewer click on the <Add Primary> button.

73. If you are Secondary Reviewer click <Add Secondary>
If you are Secondary Reviewer click <Add Secondary>. Users are now no longer constrained to have the Primary Reviewer complete the appraisal process prior to the Secondary being able to commence the appraisal for a particular paper. To illustrate this, in the screenshot below the assessment by the primary reviewer has not been completed, yet the secondary reviewer is able to commence the appraisal process.

74. A new column has been added to the form <Not Applicable>
A new column has been added to the form <Not Applicable>. Select this option if the question asked in the JBI tool is clearly not applicable to the paper being appraised. Alongside each appraisal question a text field <Comment> has also been introduced for users to insert a comment of up to 200 characters. If this field is populated with text it is maintained here in the online module for user reference. This text will not appear in any part of the review report in CReMS v.5.0.
Click <Undo> to clear all of the information entered from the screen. This will clear the entire appraisal form also, including the details added to the "Comment" field.
Each of the critical appraisal criteria must be addressed, using the Yes/No/Unclear/Not applicable radio buttons next to the criteria.

75. How to do a final assessment
It is the responsibility of the Primary Reviewer to conduct the final assessment. Clicking on the authors’ hyperlink from the studies page will open the Assessment page, showing both <Edit Primary> and <Add Final> buttons.

76. Clicking on the <Add Final> button will bring up the final assessment page which shows the results of both the primary and secondary assessments already conducted. The Primary Reviewer must consider any discrepancies between the primary and secondary assessments and, in collaboration with the Secondary Reviewer, determine the final assessment for each of the critical appraisal criteria. If agreement between the two reviewers cannot be reached, a third reviewer should be engaged to assist in the final assessment. Use the drop down box to allocate Yes/No/Undefined status in the final assessment.
The reason for excluding a study must be outlined in the final assessment.

77. Similarly, to reflect this change in process, on the main studies screen the "Status" of studies is no longer indicated by "Awaiting secondary" as in SUMARI v4.0, rather, "Primary Completed" or "Secondary Completed" will appear in the "Status" column. Please note the final assessment is still required to be completed by the Primary Reviewer; "Awaiting Final" indicates this in the "Assessment" column.

78. Screeniez next slide with drop downs active
The extraction form serves two purposes, as an online, saved, record of the details abstracted from the included paper and also to rapidly generate the appendices included in any final review report. Users will notice an asterix (*) has been included alongside certain field labels on the extraction form. For further ease of use (*) denotes which of the fields in the extraction form will be transferred to the automatically generated tables of included studies in CReMS v.5.0. To commence data extraction, click on the authors’ hyperlink of a study that shows ‘Included’ in the Status column and ‘Complete’ in the Assessment column. This will take you to the extraction details page.

79. Once the data extraction page is complete and you have clicked on <Update>, you will be taken to the New Outcome page. There are three main sections to this page, each of which will be discussed below. The New Outcome page changes slightly, depending on the source of the effectiveness data.
Single Study (same participants)

80. Outcome category
The outcome category section is a three by three matrix of all possible outcomes of an economic evaluation. The ultimate decision about the clinical and cost effectiveness of the intervention under examination is entered here, using the data extracted above on both the clinical effectiveness and cost of the intervention.
In comparing the clinical effectiveness of two alternatives there are three possibilities: (i) the intervention of interest is better or more effective (ie a ‘+’) than the comparator, (ii) the intervention is equally effective (ie a ‘0’) or (iii) the intervention is less effective (ie a ‘-’). Similarly, in terms of cost, there are three possibilities: (i) the intervention is more expensive (ie a ‘+’), (ii) the intervention and comparator’s costs are the same (ie a ‘0’), or (iii) the intervention is less expensive (ie a ‘-’). Note that each of the comparisons between intervention and comparator can only be classed as one of nine options (A – I). For example, an intervention that was shown to be more effective and less expensive would be scored as ‘G’, whereas an intervention that was less effective and of equal cost would be scored as ‘F’.

81. Once the matrix has been completed for that study, click on the <Update outcome> button and you will be taken to the results page, which lists the comparison of interest, the score from the three by three matrix (‘Dominance rating’) and the study citation.

82. New screenie to illustrate pathway
Repeat the process for all other studies that have conducted economic evaluation of the same intervention and comparator. Once this is complete, click on the ‘ACTUARI VIEW’ hyperlink on the ‘Results’ page.

83. Session 7: ACTUARI trial

84. Group Work 5: JBI ACTUARI Software Trial
JBI ACTUARI Software Trial. Workbook pages

85. Session 8: Protocol development
Allow time for participants to complete their protocols. 85

86. Session 9: Assessment
Allow 20 minutes for participants to do their MCQs then run through the answers with the group. 86

87. Session 10: Protocol Presentations
Open discussion session of participants protocols.