Evidence-Base Practice

Evidence-based practice? Decisions should be based on a combination of critical thinking and the ‘best available evidence‘.

Evidence = Various types of information outcome of scientific research, organizational facts & data, benchmarking, best practices, collective experience, personal experience, intuition

All managers base their decisions on ‘evidence’

However...

Many managers pay little or no attention to the quality of the evidence they base their decisions on

Trust me, 20 years of management experience

Teach managers how to critically evaluate the validity, and generalizability of the evidence and help them find ‘the best available’ evidence

Best available evidence  Experiential evidence: the professional insight, understanding, and expertise of practitioners  Organizational evidence; data, facts and figures, business intelligence, and benchmarks  Scientific evidence: outcome from scientific research published in peer reviewed journals  Organizational values and stakeholders’ concerns

Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process diagnosisdiagnosisinterventionintervention

Evidence based decision It’s about probabilities (not golden bullets)

Evidence-Based Practice 1991Medicine 1998Education 1999Social care, public policy 2000Nursing 2000Criminal justice ????Management?

Evidence-Based Practice

What’s the evidence for evidence-based practice?

Got evidence?  Forecasts or risk assessments based on the aggregated experience of multiple persons are more accurate than forecasts based on the experience of one person (provided that the forecasts are made independently before being averaged together) -Choudhry, N.K., et al. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005; 142 (4) -Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690 -Bauer A., et al. Forecast Evaluation with Cross Sectional Data: The Blue Chip Surveys. Economic Review, Federal Reserva bank of Atlanta, Servan-Schreiber, E., et al. Prediction Markets: Does Money Matter? Electronic Markets, 2004: 14 (31). -Scott Armstrong, J. Combining Forecasts, in Principles of Forecasting: A handbook for Researchers and Practitioners, Kluwer Academic Publishers, New York, 2001

 Professional judgments based on hard data or statistical models are more accurate than judgments solely based on experiential evidence -Yaniv, I., & Choshen-Hillel, S. (2011). Exploiting the Wisdom of Others to Make Better Decisions: Suspending Judgment Reduces Egocentrism and Increases Accuracy, Journal of Behavioral Decision Making, 2012; 25 (5) p 427–434 -Lewis, M. Moneyball: The Art of Winning an Unfair Game. Barnes and Noble, Grove, W.M. Clinical Versus Statistical Prediction. Journal of Clinical Psychology, 2005; 61(10), p 1233– 1243 Got evidence?

 A decision based on the combination of different types of evidence (experiential, organizational, scientific) leads to better outcomes than a decision based on just one source of evidence -Antman, E.M. et al, A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts, JAMA, 1992: 268 (2) p 240 – 248 -McNees, S.K. The Role of Judgment in Macroeconomic Forecasting Accuracy, International Journal of Forecasting, 1990; 6 (3), p Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690 -Tetlock, P. E. Expert Political Judgement, Princeton, NJ: Princeton University Press, 2006 Got evidence?

Evidence-Base Practice

Post mortem analysis

Evidence-based perspective NOT: Did they make the right decision? BUT: Assessment of the decision making process  What kind of evidence was there?  Was this the best available evidence?  Is there evidence from scientific research to support (or call into question) the approach taken?

Decision making process

Hospital, The Netherlands

 550 beds  3300 employees  210 medical specialists  225,000 admissions  Top Clinical & Teaching hospital  Structure: Business Units Organization

Decision: Leadership training All managers:  Board of directors  Division managers  Unit managers  Head nurses

From: 20 business units - 20 senior managers - 40 supervisors To: 20 business units – 8 senior managers - 40 supervisors Cause I: reorganization

 Ambition  Experience  Education (MBA)  Assessment  Selection Cause I: reorganization Gap

larger span of control, more responsibilities = extra skills They need extra skills We need extra skills Cause I: reorganization

Cause II: leadership climate Employee / Job satisfaction: leadership

Board of directors Cause II: leadership climate

Cause III: management development

Project team  10 members  Expertise (HR)  Representation (nurses, doctors, managers)  Support base

Process: months!  Sessions with all stakeholders (medical staff, nurses, managers, staff council) What is leadership? What is leadership? What kind of leadership does the OLVG need? What kind of leadership does the OLVG need? OLVG leadership vision

OLVG Leadership vision  Passion and business  Inspire and connect  Leadership concerns us all

Selection procedure Longlist: 30  References (other hospitals, network), reputation Shortlist: 5  Academic, Educational, Training, HR Consulting, Wild card First round: 3  Paper pitch: proposal based on documents Final round: 1  Carousel: board, medical staff, head nurses, managers

Decision making process

1.What was the problem / issue 2.What kind of evidence was there? 3.Was this the best available evidence?

1. Problem identification 2. Surfacing assumptions 3. Logic model 3 steps

For which problem is ….. the solution? -For who(m) -Why? -How big? -How do we know (what is the evidence?) Step 1: What is the problem?

Decision making process 1.What was the problem / issue 2.What kind of evidence was there? 3.Was this the best available evidence?

Assumptions are often hidden Step 2: What are the assumptions?

1. Problem identification 2. Surfacing assumptions 3. Logic model 3 steps

Logic model

Decision making process 1.What was the problem / issue? 2.What kind of evidence was there? 3.Was this the best available evidence?

Best available evidence? Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

larger span of control, more responsibilities = extra skills They need extra skills We need extra skills Cause I: reorganization

larger span of control, more responsibilities = extra skills We need extra skills Cause I: reorganization Finance? Risk management? Problem solving? Planning? Performance management? Change management? Business plan?

Cause II: leadership climate Employee / Job satisfaction: leadership

always – often – sometimes - never  I can always count on my manager when having job related problems.  I can always ask my manager for support  The relationship with my manager is good.  I have conflicts with my manager  There is a good atmosphere between my manager and me Survey: Leadership

always – often – sometimes - never  My manager inspires me to do the best in my job  My manager sets a good example  My manager makes me feel that my work is meaningful  My manager takes interest in my career development.  My manager takes interest in my career development. Survey: Leadership

Survey: outcome ‘leadership’  Favourable versus unfavourable  Comparisson with 2011  Benchmark with other divisions / units  No baseline  No confidence intervals  No focus groups  No correlations / regressions  How did new managers score?

Board of directors: strategy Cause II: leadership climate

Decision making process 1.Dus wat is het probleem? 2.Hoe zeker kunnen we er van zijn dat leiderschapstraining de oplossing voor ons probleem is? 3.Zijn er andere opties mogelijk?

Selection procedure Longlist: 30  References (other hospitals, network), reputation Shortlist: 5  Academic, Educational, Training, HR Consulting, Wild card First round: 3  Paper pitch: proposal based on documents Final round: 1  Carousel: board, medical staff, nurses, managers

leadership training: dm process  No problem definition & not objectified  No experiential evidence consulted  No organizational evidence consulted  Focus on shareholders’ concerns  Selection of training companies based on experience, recommendation or reputation  No explicit selection criteria / procedure

Postgraduate School Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process ✗ ✗ ✓ ?

Critically Appraised Topic Question: To what extent will leadership training improve the effectiveness of the leaders (e.g. managers, head nurses, physician executives) of the Amsterdam Medical Centre? In addition, what are the characteristics of effective leadership training programs?

leadership training: scientific evidence  15 meta analyses, 5 relevant  37 (‘systematic’) reviews, 2 relevant  Lots of relevant primary studies (by-catch)

leadership training: scientific evidence  Overall (50 years): small (.2) to moderate (.4) effect sizes  Little information on ROI (sometimes negative)  Tailor made programs designed on the basis of an analysis of tasks / skills are more effective and have a positive impact on motivation

leadership training: scientific evidence  Leadership trainings that focus on general management skills and / or interpersonal / social skills show higher effect sizes than those based on a specific leadership ‘model’  The trainings should be of reasonable length (>3 days) and repeated periodically to be effective

leadership training: scientific evidence  The training should include opportunities to practice (in vivo)  A working climate that supports the ‘transfer’ to the workplace (organization, supervisor, peers) motivates employees to apply what they have learned.

Reactions Who knew? Denial Anger Bargaining Acceptance

Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

Think about it: In the next weeks, before you make a decision, ask yourself  Did you ask the right questions?  What is the evidence available?  What is the quality of the evidence?  Is it the best available evidence?