Evidence-Based Practice AUPHA Annual Meeting, June 20, 2013, Monterey If doctors can do it managers can do it?

Evidence? outcome of scientific research, organizational facts & data, benchmarking, best practices, collective experience, personal experience, intuition

All managers base their decisions on ‘evidence’

However...

Many managers pay little or no attention to the quality of the evidence they base their decisions on

Trust me, 20 years of management experience

Teach managers how to critically evaluate the validity, and generalizability of the evidence and help them find ‘the best available’ evidence

Evidence-based decision

What is the added value of evidence- based practice for managers within the field of hospital care? Proof of concept

Teaching Hospital - 6 managers University Hospital - 4 managers Evidence based pilot

Phase 1: Training managers in the principles of EBP Phase 2: Examination of the current decision making processes that managers are using Phase 3: Evaluation of 4 completed projects from an EB perspective (retrospective) Phase 4: Making EB recommendations for 4 new projects (prospective) Phase 5: Evaluation Evidence based pilot

Some preliminary results

Decision making process

 Focus on procedures instead of evidence  Internal politics and power struggles  No critical appraisal of the evidence at hand  Relying on anecdotal evidence (workshops, best practices, popular management books, consultants)  One option (sometimes two)  Bias: Outcome, Halo, Confirmation, etc.

Post mortem analysis

Evidence-based perspective NOT: Did we made the right decision? BUT: Is there evidence from scientific research to support (or call into question) the approach taken? Best available evidence?

Post mortem: leadership training

leadership training: dm process  No problem definition  No organizational evidence consulted  Selection of training companies based on experience, recommendation or reputation  No explicit selection criteria / procedure  ‘Best’ presentation has won: one size fits all

leadership training: scientific evidence  15 meta analyses, 5 relevant  37 (‘systematic’) reviews, 2 relevant  Lots of relevant primary studies

leadership training: scientific evidence  Long history (30 yrs): moderate effect sizes  Senior & middle managers tend to benefit more than managers at the supervisory level  Effect on ‘poor’ leaders is limited.  Leadership trainings that focus on interpersonal / social skills show higher effect sizes than those based on a specific leadership ‘model’

Reactions Who knew? Denial Anger Bargaining Acceptance

Prospective / EB recommendations

Questions / projects  360 degree feedback  Financial incentives  Lean Six Sigma  Hand Hygiene  Goal setting  Value Based Health Care  Downsizing

Evidence-based perspective NOT: What works? BUT: What are, given the target group, the problem and the context involved, the main factors determining the success or failure of the project that need to be taken into account? Best available scientific evidence?

Prospective: Multi Source Feedback

Multi Source Feedback: background  IFMS: based on multi source feedback  Regulating bodies and insurance companies (KPI’s – prices/ revenue)  Based on CANMEDS, no standard method  New market: consulting firms

Process  Scoping session: inventory of the aspects relevant to the question  Session with leading academic  Search in relevant databases  Critical Appraisal  Summary / research synthesis  Recommendations / guidelines

Multi Source Feedback: scientific evidence  223 primary studies on MSF, 42 relevant  6 meta analyses or systematic reviews on MSF, 3 relevant  18 meta analyses or systematic reviews on ‘feedback’ or ‘performance appraisal’, 5 relevant

Content of the feedback (neg vs pos) Way of the delivery of the feedback Interpretation of the feedback Personality of the ratee Feedback orientation of the ratee Type and number of raters Selection of raters Rater reliablity (patients, nurses, colleagues) Type of response scale Development vs perfomance appraisal Organizational culture Perceived procedural justice Multi Source Feedback: main factors

Reactions Who knew? Wow! Great! Good stuff! Relevant!

Lessons learned I  New approach  Recalibrates the power dynamics (accountability!)  The profit is in the process  Different (better?) decisions were made  Doctors love it!

Lessons learned II  Hard for individual managers  It starts with the senior management team  It’s all about accountability  Support system  EBP > Planning & Control

One day, maybe … Chief Evidence Officer