1. Evidence-Base Practice

3. Evidence-based practice? Decisions should be based on a combination of critical thinking and the ‘best available evidence‘.

4. Evidence = Various types of information outcome of scientific research, organizational facts & data, benchmarking, best practices, collective experience, personal experience, intuition

5. All managers base their decisions on ‘evidence’

6. However...

7. Many managers pay little or no attention to the quality of the evidence they base their decisions on

8. Trust me, 20 years of management experience

9. Teach managers how to critically evaluate the validity, and generalizability of the evidence and help them find ‘the best available’ evidence

10. Best available evidence  Experiential evidence: the professional insight, understanding, and expertise of practitioners  Organizational evidence; data, facts and figures, business intelligence, and benchmarks  Scientific evidence: outcome from scientific research published in peer reviewed journals  Organizational values and stakeholders’ concerns

11. Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

12. Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process diagnosisdiagnosisinterventionintervention

13. Evidence based decision It’s about probabilities (not golden bullets)

14. Evidence-Based Practice 1991Medicine 1998Education 1999Social care, public policy 2000Nursing 2000Criminal justice ????Management?

15. Evidence-Based Practice

20. What’s the evidence for evidence-based practice?

21. Got evidence?  Forecasts or risk assessments based on the aggregated experience of multiple persons are more accurate than forecasts based on the experience of one person (provided that the forecasts are made independently before being averaged together) -Choudhry, N.K., et al. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005; 142 (4) -Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690 -Bauer A., et al. Forecast Evaluation with Cross Sectional Data: The Blue Chip Surveys. Economic Review, Federal Reserva bank of Atlanta, 2003. -Servan-Schreiber, E., et al. Prediction Markets: Does Money Matter? Electronic Markets, 2004: 14 (31). -Scott Armstrong, J. Combining Forecasts, in Principles of Forecasting: A handbook for Researchers and Practitioners, Kluwer Academic Publishers, New York, 2001

22.  Professional judgments based on hard data or statistical models are more accurate than judgments solely based on experiential evidence -Yaniv, I., & Choshen-Hillel, S. (2011). Exploiting the Wisdom of Others to Make Better Decisions: Suspending Judgment Reduces Egocentrism and Increases Accuracy, Journal of Behavioral Decision Making, 2012; 25 (5) p 427–434 -Lewis, M. Moneyball: The Art of Winning an Unfair Game. Barnes and Noble, 2003 -Grove, W.M. Clinical Versus Statistical Prediction. Journal of Clinical Psychology, 2005; 61(10), p 1233– 1243 Got evidence?

23.  A decision based on the combination of different types of evidence (experiential, organizational, scientific) leads to better outcomes than a decision based on just one source of evidence -Antman, E.M. et al, A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts, JAMA, 1992: 268 (2) p 240 – 248 -McNees, S.K. The Role of Judgment in Macroeconomic Forecasting Accuracy, International Journal of Forecasting, 1990; 6 (3), p 28-299 -Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690 -Tetlock, P. E. Expert Political Judgement, Princeton, NJ: Princeton University Press, 2006 Got evidence?

24. Evidence-Base Practice

25. Post mortem analysis

26. Evidence-based perspective NOT: Did they make the right decision? BUT: Assessment of the decision making process  What kind of evidence was there?  Was this the best available evidence?  Is there evidence from scientific research to support (or call into question) the approach taken?

27. Decision making process

28. Hospital, The Netherlands

29.  550 beds  3300 employees  210 medical specialists  225,000 admissions  Top Clinical & Teaching hospital  Structure: Business Units Organization

30. Decision: Leadership training All managers:  Board of directors  Division managers  Unit managers  Head nurses

31. From: 20 business units - 20 senior managers - 40 supervisors To: 20 business units – 8 senior managers - 40 supervisors Cause I: reorganization

32.  Ambition  Experience  Education (MBA)  Assessment  Selection Cause I: reorganization Gap

33. larger span of control, more responsibilities = extra skills They need extra skills We need extra skills Cause I: reorganization

34. Cause II: leadership climate Employee / Job satisfaction: leadership

35. Board of directors Cause II: leadership climate

36. Cause III: management development

37. Project team  10 members  Expertise (HR)  Representation (nurses, doctors, managers)  Support base

38. Process: months!  Sessions with all stakeholders (medical staff, nurses, managers, staff council) What is leadership? What is leadership? What kind of leadership does the OLVG need? What kind of leadership does the OLVG need? OLVG leadership vision

39. OLVG Leadership vision  Passion and business  Inspire and connect  Leadership concerns us all

40. Selection procedure Longlist: 30  References (other hospitals, network), reputation Shortlist: 5  Academic, Educational, Training, HR Consulting, Wild card First round: 3  Paper pitch: proposal based on documents Final round: 1  Carousel: board, medical staff, head nurses, managers

41. Decision making process

42. 1.What was the problem / issue 2.What kind of evidence was there? 3.Was this the best available evidence?

43. 1. Problem identification 2. Surfacing assumptions 3. Logic model 3 steps

44. For which problem is ….. the solution? -For who(m) -Why? -How big? -How do we know (what is the evidence?) Step 1: What is the problem?

45. Decision making process 1.What was the problem / issue 2.What kind of evidence was there? 3.Was this the best available evidence?

46. Assumptions are often hidden Step 2: What are the assumptions?

47. 1. Problem identification 2. Surfacing assumptions 3. Logic model 3 steps

48. Logic model

49. Decision making process 1.What was the problem / issue? 2.What kind of evidence was there? 3.Was this the best available evidence?

50. Best available evidence? Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

51. larger span of control, more responsibilities = extra skills They need extra skills We need extra skills Cause I: reorganization

52. larger span of control, more responsibilities = extra skills We need extra skills Cause I: reorganization Finance? Risk management? Problem solving? Planning? Performance management? Change management? Business plan?

53. Cause II: leadership climate Employee / Job satisfaction: leadership

54. always – often – sometimes - never  I can always count on my manager when having job related problems.  I can always ask my manager for support  The relationship with my manager is good.  I have conflicts with my manager  There is a good atmosphere between my manager and me Survey: Leadership

55. always – often – sometimes - never  My manager inspires me to do the best in my job  My manager sets a good example  My manager makes me feel that my work is meaningful  My manager takes interest in my career development.  My manager takes interest in my career development. Survey: Leadership

56. Survey: outcome ‘leadership’  Favourable versus unfavourable  Comparisson with 2011  Benchmark with other divisions / units  No baseline  No confidence intervals  No focus groups  No correlations / regressions  How did new managers score?

57. Board of directors: strategy Cause II: leadership climate

58. Decision making process 1.Dus wat is het probleem? 2.Hoe zeker kunnen we er van zijn dat leiderschapstraining de oplossing voor ons probleem is? 3.Zijn er andere opties mogelijk?

59. Selection procedure Longlist: 30  References (other hospitals, network), reputation Shortlist: 5  Academic, Educational, Training, HR Consulting, Wild card First round: 3  Paper pitch: proposal based on documents Final round: 1  Carousel: board, medical staff, nurses, managers

60. leadership training: dm process  No problem definition & not objectified  No experiential evidence consulted  No organizational evidence consulted  Focus on shareholders’ concerns  Selection of training companies based on experience, recommendation or reputation  No explicit selection criteria / procedure

61. Postgraduate School Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process ✗ ✗ ✓ ?

62. Critically Appraised Topic Question: To what extent will leadership training improve the effectiveness of the leaders (e.g. managers, head nurses, physician executives) of the Amsterdam Medical Centre? In addition, what are the characteristics of effective leadership training programs?

63. leadership training: scientific evidence  15 meta analyses, 5 relevant  37 (‘systematic’) reviews, 2 relevant  Lots of relevant primary studies (by-catch)

64. leadership training: scientific evidence  Overall (50 years): small (.2) to moderate (.4) effect sizes  Little information on ROI (sometimes negative)  Tailor made programs designed on the basis of an analysis of tasks / skills are more effective and have a positive impact on motivation

65. leadership training: scientific evidence  Leadership trainings that focus on general management skills and / or interpersonal / social skills show higher effect sizes than those based on a specific leadership ‘model’  The trainings should be of reasonable length (>3 days) and repeated periodically to be effective

66. leadership training: scientific evidence  The training should include opportunities to practice (in vivo)  A working climate that supports the ‘transfer’ to the workplace (organization, supervisor, peers) motivates employees to apply what they have learned.

67. Reactions Who knew? Denial Anger Bargaining Acceptance

68. Evidence based decision Best available experiential evidence Best available organizational evidence Organizational values and stakeholders’ concerns Best available scientific evidence Decision making process

69. Think about it: In the next weeks, before you make a decision, ask yourself  Did you ask the right questions?  What is the evidence available?  What is the quality of the evidence?  Is it the best available evidence?