1. The Alberta Health Technologies Decision Process: Post Policy Implementation Review Presenter: Sarah Flynn Authors: Dr. Anderson Chuck, Institute of Health Economics; Sarah Flynn, Alberta Health; Dr. Nina Buscemi, Alberta Health; Dr. Kathryn Ambler, Alberta Health

2. Disclosure I have no actual or potential conflict of interest in relation to this topic or presentation. 2

3. The Alberta Health Technologies Decision Process (AHTDP) Provincial review process for health technologies and services that provides evidence and information to inform decision-making A collaboration with Health Technology Assessment Partners. Reviews consider: –effectiveness –safety –cost-effectiveness and budget impact –ethical and legal implications –patient and provider perspectives –Potential policy approaches and implications 3

4. How do we assess impact? Health care decisions may be influenced by clinical impact, cost-effectiveness findings. Limited opportunities to actively monitor implementation of AHTDP-informed policy decisions. Monitoring and evaluation by implementer (typically AHS). 4

5. Post-Policy Implementation Review (PPIR) PPIR is a review of a policy decision to determine: PPIR offers evaluation, accountability 5 Did the policy achieve the desired results? Were costs and benefits in line with expectations? Were there any unintended or unforeseen consequences? What are learnings for future policy development? Prospective Retrospective

6. Retrospective PPIR 6 Key prerequisites for PPIR: –Commitment of all participants –Access to information and data –Readiness to apply the findings Some policies are better candidates than others –Ideal conditions for PPIR –Policy implemented 2 – 5 years ago –Clear and logical policy goals –Adequate studies/evidence available –Access to original implementers

7. PPIR Framework 7

8. Testing for Preterm Labour Preterm labour/delivery occurs between 20 – 37 wks Leading cause of neonatal mortality and morbidity Contributes to neurodevelopmental problems, respiratory/pulmonary dysfunction, hearing and visual impairment, and other long-term health problems. Interventions are available to reduce morbidity/mortality Issue: –Not all symptomatic women will deliver –Some women experiencing false labour are being transferred and admitted to hospital Solution: A test which is good at identifying false labour The TLi IQ ® System or fetal fibronectin (fFN) testing 8

9. 2006 AHDTP Review 9 The fFN test was reviewed through the AHTDP to determine if it should be publicly funded. Finding: –fFN test can aid in ruling out unnecessary interventions for women in false labour. –fFN testing would result in cost savings to the provincial health system through the avoidance of ambulance transfers and decreased length of stay.

10. The Policy Decision (2006) 10 Introduce fFN testing for preterm labour as a publicly funded service available to all Alberta women by 2008 –Regional Health Authorities (RHAs) to implement their preferred service delivery models and tests. –Funding from existing budget allocations –RHAs encouraged to implement quality assurance mechanisms, guidelines and standards. 2008 amendment allowed RHAs to chose between fFN and an alternate test (Actim™ Partus)

11. PPIR Methodology 11 Is there new evidence on performance differences between fFN and Actim™ Partus? Literature Review Update How was implementation conducted? Barriers, facilitators, and unintended consequences Key Informant Interviews Health system impact and costs. Transfers, length of stay and admissions Economic Analysis Institute of Health Economics conducted a 2015 retrospective PPIR:

12. Do performance differences exist between the two tests? Both the fFN test and the Actim™ Partus test had high specificity and negative predictive values. Both the fFN test and the Actim™ Partus test had low sensitivity and positive predictive values. FFN was more accurate at predicting preterm delivery than Actim™ Partus. No evidence to suggest that the system adopted in Alberta (fFN testing) should be changed based on diagnostic performance 12

13. Was PTL testing implemented according to the 2006 policy? All Regional Health Authorities (RHAs) fully implemented testing by 2008, using the same test (fetal fibronectin).  RHAs covered costs through existing budgets* Staff trained using a variety of materials Providers generally trust the fFN test and consider test results as part of their routine for managing patients with PTL symptoms  Most of the training occurred at policy implementation; levels of training varied  Access to testing equipment varied 13

14. Has PTL testing reduced unnecessary hospital transfers, admissions, and length of stay? Physicians appear to place more significance on positive test results when deciding to transfer or admit women (inappropriate use of test). Testing did not reduce unnecessary ambulance transfers or admissions (women in false labour). –Ambulance transfers In true labour. 1.91 / 7.45 times more likely to transfer if +/- In false labour: 0.78 / 2.22 times more likely to transfer +/- –Admissions In true labour. 1.68 / 0.44 times more likely to transfer if +/- In false labour: 5.38 / 0.47 times more likely to transfer +/- 14

15. Has PTL testing had any unintended consequences? More patients being transferred or admitted, regardless of test results fFN testing did not reduce unnecessary ambulance transfers or admissions for women in false PTL Testing has the number of appropriate ambulance transfers and admissions for preterm pregnancies in true labour. 15

16. Has PTL testing resulted in net cost savings for the system? Testing did not reduce unnecessary use of healthcare services, and associated costs. Total health system costs increased due to test purchases and increased appropriate care due to testing (more women in true PTL receiving care). The increase in health service utilization resulting from testing has cost the health system an estimated $3,458,443 for appropriate utilization and $730,724 for unnecessary utilization 16

17. What next? Shared results with the key provincial stakeholder group: –The Maternal, Newborn, Child and Youth (MNCY) Strategic Clinical Network (SCN) Feedback gathered The Alberta Advisory Committee on Health Technologies made a formal recommendation to the MNCY SCN: –Do not maintain the status quo –Consider the PPIR and Policy Analysis results –Develop a plan of action to address inappropriate use of preterm labour testing. March 20 decision to discontinue testing 17

18. Reflections 18 First PPIR of an AHTDP Decision: Challenges of a retrospective approach Working from an AHTDP review Capitalizing on HTA competencies Lessons for future reviews

19. Lessons 19 Strengthening AHTDP reviews: –Draw on HTA findings to develop comprehensive policy options and recommendations –Leverage HTA findings to support strategic, measured implementation Strengthening PPIR: –Identify a key stakeholder or “client” –Prospectively evaluate or review policy decisions, where possible