1. Final Canadian National Delphi Consensus Results - What Are The Appropriate National Clinical Pharmacy Key Performance Indicators (cpKPI) For Canadian Hospital Pharmacists?
  Olavo Fernandes BScPhm, ACPR, PharmD, FCSHP
Director of Pharmacy- Clinical, University Health Network, Toronto ON
Assistant Professor (Status)- Leslie Dan Faculty of Pharmacy
Sean K. Gorman, BSc(Pharm), ACPR, PharmD
Regional Coordinator - Clinical Quality and Research, Pharmacotherapeutic Specialist – Critical Care  Interior Health Authority, Clinical Associate Professor – Faculty of Pharmaceutical Sciences, UBC
Kent Toombs BSc(Pharm), ACPR
Clinical Pharmacy Manager, Capital District Health Authority, Halifax, NS
Canadian Hospital Pharmacy Leadership Conference , June 8, 2013

2. Objectives
To outline the key elements of the national consensus process in developing clinical pharmacy key performance indicators (cpKPI) for hospital pharmacists
including consensus definition, selection criteria for cpKPI,  critical topic/ activity foci and pre-Delphi candidate cpKPI)
To report the final results of the recent national Delphi consensus phase to establish a final suite of cpKPI
To summarize the next phases and communication plans in the national cpKPI process :
exploring interprofessional/ external stakeholder feedback,
national information capture/ measurement systems,
cpKPI knowledge translation kit
practical definition and measurement questions
pan-Canadian communication/ Manuscript publications / posters

3. Overall Goal of the National cpKPI Collaborative / National Consensus Process
To develop a core set of national clinical pharmacy KPI for hospital pharmacists via a systematic national evidence-informed consensus process 3

4. Key Performance Indicators (KPI)
What is it?
“Quantifiable measures that reflect the critical success factors of an organization” 1
Quantitative measures of quality
Why is it important?
Elevate professional accountability & transparency
Serve to improve quality of care
1. Doucette D, Millen B. Should Key Performance Indicators for Clinical Pharmacy Services Be Mandatory, Can J Hosp Pharm 2011; 64(1):55-57. 4

5. Rationale for clinical pharmacy KPI (cpKPI)
GAP: currently NO established national or international consensus on what constitutes a KPI for clinical pharmacy services
Rationale: To advance practice toward desired evidence-informed patient outcomes
cpKPI will serve to better define minimum standards and permit benchmark comparisons within and between organizations

6. National cpKPI Collaborative Definition of cpKPI
Five pillars/ characteristics of cpKPI:
Reflect a desired quality practice and
A metric with a link to direct patient care and
Link to evidence of impact on meaningful patient outcomes and
A pharmacy/ pharmacist sensitive metric
Feasible to measure
Clinical metric would have to fulfill all 5 pillars to qualify as a candidate cpKPI

7. Hierarchy of Study Outcomes (AHRQ)
Level 1: Clinical and QoL outcomes
Morbidity, mortality, adverse events
Level 2 : Surrogate outcomes
I.e. blood glucose, blood pressure, cholesterol
Level 3: Measureable variables with an indirect or unestablished connection to target outcome
I.e. medication disease state knowledge
Level 4: Indirect variables
I.e. patient satisfaction, “potential adverse events”

8. Pharmaceutical care patient care process
Should Align with Local Consensus or Guidelines for Prioritization of Hospital Pharmacist Activities
6 Domains
Pharmaceutical care patient care process
Operational patient care supporting activities
Drug information
Teaching/Education/Learning
Research
Service (clinical and pharmacy committees)
*Extracted from UHN Pharmacist Pyramid-Prioritization of Pharmacist Activities Draft

9. Information Gathering - Prior to Consensus Building
Optimal National cpKPI
Literature:
1.Evidence
2.Process
CSHP 2015/ CPhA Blueprint
Front-line Staff/Leaders
Peer Hospital Best Practices
Pharmacy Leadership
Information Gathering - Prior to Consensus Building

10. Information Gathering
Proposed Timeline
CSHP endorsed concept
Pre-Delphi
Delphi
Post-Delphi
KPIWG formed
Delphi Process
Information Gathering
May 2011
Dec’12-Mar’13
We are here
Aug 2011
Feb 2013
Survey Development
Consensus Meeting

11. Key National Process Milestones
National consensus definition – cpKPI (Aug 2011)
National Crude Inventory of candidate cpKPI / metrics (started Jan 2012)
National Information-gathering Process: Workshops/ Information sessions-Front line feedback (Feb Nov 2012)
Outcome and Process Debates/ Finalized Evidence summary tables (June-July 2012)
A priori consensus cpKPI selection criteria (ideal attributes)– “Slavik 11” (Finalized July 2012)
Key cpKPI Critical Activity / Topic Areas – “Doucette 8” (Finalized August 2012)
Final Pre-Delphi Candidate cpKPI list (October 2012)
Selection of National Delphi Panel members (November 2012)
Delphi Panel Process – Round 1-3 (Dec 21, Mar )
cpKPI Live Meeting (February 5, 2013)

12. Slavik -11- Consensus Criteria – Ideal Attributes
 Based on high quality literature evidence
(e.g. Observational data vs. RCT vs. systematic review)
Relevant impact on clinically important outcomes
(e.g. Surrogate versus clinical endpoints, effect size of intervention)
Best-suited to pharmacist’s role
(e.g. Identifies pharmacist-specific clinical role vs. GP vs. RN)
Attributable to direct patient care
(e.g. Marker of clinical intervention, not distribution)
Specific to pharmaceutical care process
(e.g. Related to generally-accepted PC processes)
Aligned with professional goals, objectives, practices
(e.g. Accreditation Canada ROPs, standards, CSHP Vision 2015, etc.)

13. Slavik -11- Consensus Criteria – Ideal Attributes
Accepted disease-based quality indicator
(e.g. ACEI or BB for HF, VTE prophylaxis in hospitalized patients)
Feasible to measure
(e.g. Reliable measurement systems can/could be put in place)
Efficient to measure
(E.g. Acceptable time commitment, useable)
Valuable quality measure
(E.g. Prevalent, impactful problem with practical, proven interventions)
Generalizability
(E.g. Versatile enough to be applied in large, academic and small community sites)

14. Delphi panelist priority ranking of consensus cpKPI selection criteria- Final – Mar 2013

15. Bringing the evidence all together with extrapolation………
RCT Outcome Findings
Gillespie U et al RCT
Integrated Intervention pharmaceutical care Integrated Intervention
post-discharge hospital visits (ED + readmissions)
emergency department visits
drug related readmissions
Makowsky MJ et al RCT
1. “overall quality score”
2. 3 and 6 month all-cause readmission
(hospital or ED visit after index hospital admission)
Chisholm-Burns MA et al 2010, systematic review w/ focussed meta-analyses
HbA1c , LDL Cholesterol, Blood Pressure
Adverse Drug Events
Bond et. al. (2007) Observational Study
Clinical Pharmacy & Mortality
admission drug histories
medical rounds participation
CPR team participation
Kaboli PJ et al. (2006) Systematic Review
attendance on patient care rounds
patient interviews and assessments
medication reconciliation
discharge “counselling” (patient medication education)
follow-up after discharge

16. Evidence Summary Tables
Discussion: specific group suggestions to modify or concur with the follow sections
Strengths and Limitations
Application/Synthesis: How does this study inform the cpKPI selection process  (methods, cpKPI selection criteria, and candidate cpKPI)?
What are the patterns (similarities and differences) compared to other key papers?
Purpose: August- used to refresh and focus outcome evidence for streamlining ; Nov-  Used by Delphi panelists to support ranking and decision making

17. Levels of Evidence
Observational Studies
Systematic Reviews
Randomized Controlled Trials

18. PRACTICE QUESTION
Does pharmacist-led comprehensive pharmaceutical care reduce morbidity (& other meaningful patient outcomes) for elderly hospitalized patients?

19. A Comprehensive Pharmacist Intervention to Reduce Morbidity in Patients 80 years or Older Gillespie U et al. Arch Intern Med 2009; 169(9):
Objectives:
assess the effectiveness of interventions performed by ward-based pharmacists on morbidity and overall use of (secondary) hospital care
Design:
prospective, single centre, unblinded, randomized control trial
patient- unit of randomization, central centre
Setting: 2 acute internal medicine wards (university teaching hospital) in Uppsala, Sweden
Duration: Oct 2005-June 2006
Patients:
Patients 80 years or older admitted to 2 acute care internal medicine wards
Written informed consent
Sample size calculation : 400 patients

20. A Comprehensive Pharmacist Intervention to Reduce Morbidity in Patients 80 years or Older Gillespie U et al. Arch Intern Med 2009; 169(9):
Patients randomized to:
intervention (comprehensive care by hospital pharmacist)
Ward based clinical pharmacists
comprehensive patient interview, BPMH, admission medication reconciliation,
pharmaceutical care drug review (Cipolle method) to identify and resolve DTPs, physician interventions on drug selection, dosages, monitoring….
Addressed: indication, effectiveness, safety and adherence
DTPs discussed on ward rounds
Patients received education and discharge counselling/ reconciliation
pharmacist discharge letter communicated to primary care physicians by pharmacists
Follow up telephone call 2 months after discharge
control: standard care without pharmacist involvement by physicians and nurses

21. Results: Major Outcomes Gillespie U et al
Results: Major Outcomes Gillespie U et al. Arch Intern Med 2009; 169(9):
Patients Evaluated (n=368, 182 intervention / 186 control)
over a 12 month period
Post-Discharge Hospital Visits (ED + readmission)
↓ 16% intervention group
(quotient 1.88 vs. 2.24, 95% CI )
Emergency Department Visits:
↓ 47% intervention group
(quotient 0.35 vs. 0.66, 95% CI )
Drug Related Readmissions:
↓ 80% intervention group
(quotient 0.06 vs. 0.32, 95% CI )
Aside: Balancing Measures- Readmissions Alone and Mortality :
- No significant difference

22. Bringing the evidence all together with extrapolation………
RCT Outcome Findings
Gillespie U et al RCT
Integrated Intervention pharmaceutical care Integrated Intervention
post-discharge hospital visits (ED + readmissions)
emergency department visits
drug related readmissions
Makowsky MJ et al RCT
1. “overall quality score”
2. 3 and 6 month all-cause readmission
(hospital or ED visit after index hospital admission)
Chisholm-Burns MA et al 2010, systematic review w/ focussed meta-analyses
HbA1c , LDL Cholesterol, Blood Pressure
Adverse Drug Events
Bond et. al. (2007) Observational Study
Clinical Pharmacy & Mortality
admission drug histories
medical rounds participation
CPR team participation
Kaboli PJ et al. (2006) Systematic Review
attendance on patient care rounds
patient interviews and assessments
medication reconciliation
discharge “counselling” (patient medication education)
follow-up after discharge

23. Doucette 8- Consensus Critical Activity / Topic Areas
Pharmaceutical Care – Integrated (DTP assessment/ care plan/ monitoring)
Medication Reconciliation- BPMH/Med History Taking
Medication Reconciliation- Admission Reconciliation
Medication Reconciliation- Discharge Reconciliation
Team (or Patient) Rounds
Discharge Patient Education / Counselling
Post Discharge Follow-Up
Disease or Drug Specific – Best Practice Quality Indicators

24. Doucette 8- Consensus Critical Activity / Topic Areas
Dot voting: 20 dots per person
Assign proportionately
Question:
Will measuring a cpKPI in this “critical activity topic area” be useful to advance clinical pharmacy practice to improve the quality of patient care?
Semchuk-26 Draft Candidate KPI list

25. DEMOGRAPHICS OF cpKPI DELPHI PANEL

26. What is your practice setting (check all that apply)?
69% (18)
Other= Administration, Pharmacy Association, Oversee multiple sites, Regional Health Authority, Long Term Care and Rehabilitation Centre, District health authority with tertiary and community practice.
35% (9)
27% (7)
27% (7)
23% (6)
12% (3)
Community hospital
Teaching hospital
Tertiary care hospital
Academia
Other
Clinic

27. Do you work primarily with pediatrics or adults?
88% (23)
12% (3)
Pediatrics

28. How many years of experience do you have as a licensed Pharmacist?
65% (17)
19% (5)
8% (2)
8% (2)
0% (0)
0-5 years
6-10 years
11-15 years
16-20 years
20+ years

29. What is your educational background?
Other: MBA, BSc (Pharmacology), EXTRA Fellowship (CFHI) Certified Health Executive (CHE), MBA, Post PharmD Residency, Certified Geriatric Pharmacist
100% (26)
54% (14)
54% (14)
23% (6)
19% (5)
Residency (ACPR)
Masters Degree
BScPhm
PharmD
Other

30. DOUCETTE 8 – 20 Dot Voting RESULTS

31. Doucette 8- Consensus Critical Activity / Topic Areas
Pharmaceutical Care – Integrated (DTP assessment/ care plan/ monitoring)
Medication Reconciliation- BPMH/Med History Taking
Medication Reconciliation- Admission Reconciliation
Medication Reconciliation- Discharge Reconciliation
Interprofessional (team) patient care rounds
Discharge Patient Education / Counselling
Post Discharge Follow-Up
Disease or Drug Specific – Best Practice Quality Indicators
Used to create “Semchuk 26” candidate cpKPI list

33. Grape Analogy: BUNDLES AND CRITICAL ELEMENTS

34. Modified Delphi Process Methodology
A Delphi technique is a structured process commonly used to develop consensus healthcare quality indicators
It was developed to minimize influence from more vocal group members, and utilizes surveys or questionnaires instead of discussion.
frequently used with expert panels to generate consensus on healthcare issues
To arrive at consensus, a modified Delphi technique will be used.
This ‘modified” technique is an iterative process that builds consensus using three rounds of anonymous panelist ratings with a live/tcon meeting

35. Delphi Rounds Standardized Orientation Round 1 Round 2
Audio PowerPoint + Mandatory Pre-Reading
Round 1
Demographic Information; Panelist ranks Semchuk 26 cpKPI, For each Slavik 11 and Overall Ranking, Suggest new cpKPI
Round 2
Review R1 aggregate summary/ report card for each cpKPI
Frequency Graphs Summary
Review anonymous qualitative comments
Panelist re-ranks all cpKPI
Live Meeting – Debate and Discussion to inform individual rankings
identify meet other panelists for the first time
Round 3
Review Feb 5 Live Minutes , R2 summaries (as above), Final Rankings

36. Delphi Rounds Individual cpKPI ratings
Opportunity to suggest additional candidate cpKPI (round 1 only to allow panel feedback)
Ranking of priority of “Doucette 8” Critical Activities and “Slavik 11” Selection Criteria
Combining cpKPI, Creating New cpKPI by modifying working (ie cpKPI 27, 28, 30)
Threshold for consensus consideration:
75% of panelists assign a rating of 7-9 on the 9 point Likert scale
MAGIC NUMBER = 20

37. Round 1 Qualitative Panelist Discussion Themes
Qualitative discussion themes while comparing cpKPI included:
Varying degrees of sensitivity to pharmacists’ contribution
Varying degrees of feasibility of measurement
Varying degrees of generalizability across practice areas (i.e.. psychiatry, surgery) as well as across different types of hospitals (i.e. urban versus rural)
Inter-relationships between: medication reconciliation cpKPIs; discharge/ inpatient counselling cpKPIs

38. Round 1 – 3 New cpKPI Submitted by Panelists
cpKPI #27: combined pharmacist admission Med Rec + BPMH Number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (includes a pharmacist-BPMH OR pharmacist-BPMH-review as part of reconciliation as well as resolution of identified discrepancies).
cpKPI #28: Proactive bundle; Number (or proportion) of patients receiving “proactive comprehensive, direct patient care by a pharmacist in collaboration with the health care team” (Makowsky Collaborate RCT Bundle).
cpKPI #29: Time on Ward Committed decentralized clinical pharmacist time per patient day per patient service.

39. Live Feb 5: Meeting Issues and Controversies
How to optimally handle process of care vs. disease/drug-specific indicators?
High Value Action “DTP resolved” as a subset
Grape Theory: Bundles and Critical Elements
Number vs. proportion
A priori Suite properties
“High Risk vs. All Patients”

40. Final Delphi Results 8 cpKPI have officially met consensus
Round 3 Final Rankings
8 cpKPI have officially met consensus
6/8 Doucette Categories represented with combos  

41. Final 8: cpKPI Number and Description
Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist (#11)
Number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (combined BPMH) (#27)
Number (or proportion) of patients for whom clinical pharmacists have completed (executed/implemented) a pharmaceutical care plan (#27)
Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management
Sean K. Gorman, PharmD 2010

42. Number of total drug therapy problems (DTPs) resolved by pharmacists
Final 8: cpKPI Number and Description
Number of total drug therapy problems (DTPs) resolved by pharmacists
Number (or proportion) of patients receiving "proactive comprehensive, direct patient care by a pharmacist in collaboration with the health care team" (Makowsky Collaborate RCT Proactive Bundle) (#28)
Number (or proportion) of hospitalized patients who receive medication counselling by a pharmacist at discharge
Number (or proportion) of patients who have received in person education from a pharmacist about their disease(s) and medication(s) during their hospital stay
Sean K. Gorman, PharmD 2010

43. How do the final national clinical pharmacy key performance indicators align with national consensus selection criteria?

44. National cpKPI Collaborative Next Steps
7 Post-Delphi Phases
cpKPI knowledge translation kit- practical getting started kit
Final 8 cpKPI-specific measurement summaries, background, 7 step change management framework
Final 8 : Practical Outstanding Questions
Wording, outstanding questions, practical definitions, practical measurement issues
Exploring external stakeholder feedback
Interprofessional : physicians, nurses, Ministry of Health, pharmacists- US, UK, NZ, Aus, patients, hospital administrators
National information capture systems / measurement systems (“apps”)
Pan-Canadian Communication of Final Delphi Results
2 Manuscript Publications/ 7 Conference Abstracts
Formal “Pilot” Sites