1. Pit stop EBMgt Evidence-Based Management (Searching & Finding Scientific Evidence) Pitstop Evidence-Based HR, VOV lerend netwerk, Antwerpen, 11 januari 2013

2. Pit stop EBMgt 1. Evidence-Based Management A (very) Short Introduction

3. Pit stop EBMgt McMaster University Medical School, Canada Medicine: Founding fathers David Sackett Gordon Guyatt

4. Pit stop EBMgt How it all started

5. Pit stop EBMgt  More than 1 million articles in 40,000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a GP more than 100!)  Most of the new insights and treatment methods don’t reach the target group Problem I: too much information

6. Pit stop EBMgt Problem I: too much information  HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR manager to keep up this means reading 3 to 4 articles every day (for a ‘general’ manager more than 50!)

7. Pit stop EBMgt Problem II: persistent convictions if you’re hyperventilating breathe into a bag

8. Pit stop EBMgt Problem III: jumping to conclusions elderly people who have an irregular heartbeat are much more likely to die of coronary disease give them a drug that reduces the number of irregular beats

9. Pit stop EBMgt How 40,000 cardiologists can be wrong In the early 1980s newly introduced anti- arrhythmic drugs were found to be highly successful at suppressing arrhythmias. Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality. The CAST trial revealed Excess mortality of 56/1000. By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.

10. Pit stop EBMgt David Sackett  Half of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half.  The most important thing to learn is how to learn on your own.  (Remember that your teachers are as full of bullshit as your parents)

11. Pit stop EBMgt Evidence-Based Practice 1991Medicine 1998Education 1999Social care 2000Nursing 2000Criminal justice ????Management?

12. Pit stop EBMgt 2. Evidence based management: What is it?

13. Pit stop EBMgt Evidence based practice: Improve information to support decision making Definition

14. Pit stop EBMgt Definition Evidence-based management means making decisions about the management of employees, teams or organizations through the conscientious, explicit and judicious use of four sources of information: 1. The best available scientific evidence 2. Organizational facts, metrics and characteristics 3. Stakeholders’ values and concerns 4. Practitioner expertise and judgment

15. Pit stop EBMgt Four sources

16. Pit stop EBMgt Evidence-based practice:  Focuses on the decision making process  Thinks in terms of probability (instead of golden bullets).

17. Pit stop EBMgt 3. Evidence-based management: Why do we need it?

18. Pit stop EBMgt Four sources

19. Pit stop EBMgt Trust me: 20 years of management experience!

20. Pit stop EBMgt Bounded rationality

21. Pit stop EBMgt Bounded rationality System 1  Fast  Intuitive, associative  heuristics & biases System 2  Slow (lazy)  Deliberate, ‘reasoning’  Rational

22. Pit stop EBMgt  Seeing order in randomness  Mental corner cutting  Misinterpretation of incomplete data  Halo effect  False consensus effect  Group think  Self serving bias  Sunk cost fallacy  Cognitive dissonance reduction System 1: prone to biases  Confirmation bias  Authority bias  Small numbers fallacy  In-group bias  Recall bias  Anchoring bias  Inaccurate covariation detection  Distortions due to plausibility

23. Pit stop EBMgt 2. Searching & Finding The Best Available Scientific Evidence

24. Pit stop EBMgt Step 1: Formulate a focused question

25. Pit stop EBMgt 5-step approach EBMgt is a 5-step approach 1.Formulate an answerable question (PICOC) 2.Search for the best available evidence 3.Critical appraise the quality of the found evidence 4.Integrate the evidence with managerial expertise and organizational concerns and apply 5.Monitor and evaluate the results

26. Pit stop EBMgt Asking the right question? Effect vs Non-effect

27. Pit stop EBMgt Types of questions: effect Does it work? Does it work better than....? Does it have an effect on....? What is the success factor for....? What is required to make it work...? Will it do more good than harm? Effect

28. Pit stop EBMgt Types of questions: non-effect Needs:What do people want or need? Attitude:What do people think or feel? Experience: What are peoples’ experiences? Prevalence:How many / often do people / organizations...? Procedure: How can we implement...? Process:How does it work? Explanation:Why / how does it work? Economics:How much does it cost?

29. Pit stop EBMgt Focused question?  Does team-building work?  What are the costs and benefits of self-steering teams?  What are the success factors for culture change?  Does management development improve the performance of managers?  Does employee participation prevent resistance to change?  How do employees feel about 360 degree feedback?

30. Pit stop EBMgt  What is a ‘team’?  What kind of teams?  In what contexts/settings?  What counts as ‘team-building’?  What does ‘work’ mean?  What outcomes are relevant?  Over what time periods? Foreground question?  Does team-building work?

31. Pit stop EBMgt P =Population I = Intervention (or success factor) C = Comparison O = Outcome C = Context Answerable question: PICOC

32. Pit stop EBMgt Example: merger

33. Pit stop EBMgt Answerable question: PICOC Imagine you are a consultant, your client is the board of directors of a large Canadian health-care organization. The board of directors has plans for a merger with a smaller healthcare organization in a nearby town. However, it’s been said that the organizational culture differs widely between the two organizations. The board of directors asks you if this culture-difference can impede a successful outcome of a merger. Most of them intuitively sense that cultural differences matter, but they want an evidence-based advice.

34. Pit stop EBMgt Answerable question: PICOC What else would you like to know?

35. Pit stop EBMgt Answerable question: PICOC P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?

36. Pit stop EBMgt According to the board the objective of the merger is to integrate the back-office of the two organizations (ICT, finance, purchasing, facilities, personnel administration, etc.) in order to create economy of scale. The front offices and primary process of the two organizations will remain separate. The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations). Answerable question: PICOC

37. Pit stop EBMgt P =back office employees in a healthcare organisation I = merger, integration back office C = status quo O = economy of scale C = different organizational culture, unequal Answerable question: PICOC

38. Pit stop EBMgt 1.Read the following five scenario’s 2.Formulate on the basis of each scenario a focused question (use the PICOC format). Exercise

39. Pit stop EBMgt Scenario 2 “Ik werk als manager bij een grote financiële dienstverlener. Het concern is een holdingmaatschappij met daaronder een aantal divisies met een eigen P&L verantwoordelijkheid. Het accountmanagement van de divisies wordt aangestuurd op basis van een eigen set kritische prestatie indicatoren (kpi’s) die niet onderling is afgestemd. Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen rijden bij gezamenlijke klanten en de omzet wordt ‘weggekaapt’ bij de ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de Executive Board van de holding een beroep gedaan op de divisiedirecties om meer met elkaar te gaan samenwerken. Mijn vraag is: staat het feit dat de divisies geen gedeelde kpi’s hebben een goede interne samenwerking in de weg of hoeft dit geen probleem te zijn?”

40. Pit stop EBMgt Scenario 3 “I am a Dutch quality manager at a large international brewery. In the past 4 years I facilitated a lean management program that is running for more than 10 years in this organization. Since a couple of years, however, we see a decline in results, such as:  The number of problem analysis and improvement teams has been increased, but the amount of breakdowns and short stops in the production lines has not decreased.  There is slow progress for the total program for the Netherlands at this moment. According to the middle managers the improvement program does not address really important topics. I would therefore like to know how our employees feel about lean management.”

41. Pit stop EBMgt 1.Neem een actueel probleem / vraag / issue dat speelt in de organisatie waar je stage hebt gelopen (of een vraagstuk dat je persoonlijke interesse heeft). 2.Formuleer op basis daarvan een onderzoeksvraag 3.Geef aan wat voor type vraag het is (effect of non-effect) 4.Benoem de PICOC elementen.

42. Pit stop EBMgt Step 2: Search for the best available evidence

43. Pit stop EBMgt 5-step approach EBMgt is a 5-step approach 1.Formulate an answerable question (PICOC) 2.Search for the best available evidence 3.Critical appraise the quality of the found evidence 4.Integrate the evidence with managerial expertise and organizational concerns and apply 5.Monitor and evaluate the results

44. Pit stop EBMgt Searching evidence Searching evidence What do we search?

45. Pit stop EBMgt Type of Information Source Current Information Wall street Journal, Financial Times, Business week, Financieel Dagblad Overview of a subject General background Academic Information Statistical Information Textbooks and popular booksEncyclopedias, yearbooks & book reviews ABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science Direct CBS Statline, Eurostat Theories about a subject Textbooks and encyclopedias Information sources Company information Company Annual Reports, Datastream, Factiva.com, Amadeus

46. Pit stop EBMgt Articles in peer reviewed journals

47. Postgraduate Course Searching for evidence

48. Databases Postgraduate Course  ABI/INFORM  Business Source Elite  PsycINFO  Web of Knowledge  ERIC  Google Scholar

49. Postgraduate Course Searching for evidence

50. Searching evidence Searching evidence How do we search? Search Strategy Postgraduate Course

51. Pit stop EBMgt Two types of search strategies Search strategy Building blocks method Snowball method

52. Pit stop EBMgt Snowball method Starting from one book or article, you search for other literature on the same topic.  Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article).  Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).

53. Pit stop EBMgt Snowball method

54. Pit stop EBMgt Synonyms or related terms …. Synonyms or related terms …. Synonyms or related terms …. Building blocks method Synonyms or related terms …. Keyword 1Keyword 2Keyword 3Keyword 4 ANDANDAND OROROR

55. Pit stop EBMgt P =back office employees, health care I = merger, integration, back office C = status quo O = economy of scale C = different organizational culture, unequal Answerable question: PICOC 1.Underline the keywords 2.Number the order of importance from 1-4

56. Pit stop EBMgt P =back office employees 5. healthcare I = 1. merger, 3. integration, back office C = status quo O = 4. economy of scale C = different 2. organizational culture, unequal Answerable question: PICOC 1.Underline the keywords 2.Number the order of importance

57. Pit stop EBMgt Search terms Search terms Operationalise your Pico elements! O = long term profitability?

58. Pit stop EBMgt Construct validity The Ambidextrous Organization (??)

59. Pit stop EBMgt corporate culture: organizational behavior/character, corporate identity merger: acquisition, take-over, fusion, combination, unification profitability: profit, advantage, return on investment, shareholder value The keywords of your PICOC may be enough. If not, select more words by using: Select keywords Select keywords  synonyms  alternate spelling, translations  related terms / words / subjects  narrower or broader terms

60. Pit stop EBMgt Search in the following order Search in the following order 1. Thesaurus (SU) 2. Title (TI) 3. Abstract (AB) 4. Alternate spellings, synonyms

61. Pit stop EBMgt Merger Fusion Combination Take over Acquisition Unification … 1. Merger 3. Integration Healthcare organization Non profit Not for profit 4. Health care organization AND Search Query: an example Search Query: an example IIO OROR Integration Corporate culture Organizational behavior Organizational character Corporate identity Core beliefs Shared values 2. Corporate culture C AND OR

62. Pit stop EBMgt oranges apples apples AND oranges apples OR oranges oranges apples Boolean operators

63. Pit stop EBMgt Start up Start up

64. Pit stop EBMgt Always search for indexed topics or thesaurus terms Always search for indexed topics or thesaurus terms

65. Pit stop EBMgt thesaurus terms thesaurus terms

66. Pit stop EBMgt Use the history function to combine results Combining results Combining results

67. Pit stop EBMgt Use the history function to combine results Combining results Combining results

68. Pit stop EBMgt Narrow your results Narrow your results

69. Pit stop EBMgt Narrow your results Narrow your results

70. Pit stop EBMgt Save your results Save your results

71. Pit stop EBMgt If an article is not available in ABI/ Inform or BSE: try Google (advanced search) If an article is not available in ABI/ Inform or BSE: try Google (advanced search)

72. Pit stop EBMgt Summary 1.Formulate a focused question and break it down into keywords (PICOC), 2.Underline the 2 to 4 most relevant keywords, 3.Start with the most relevant keyword: is it listed in the thesaurus? If not: search in the title or abstract. 4.Repeat your search with alternative terms, synonyms or alternate spellings. 5.Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative terms) with OR (use the history function!) 6.Repeat above steps for keyword 2 7.Combine the outcome of your search with keyword 1 and 2 with AND (use the history function!) 8.Filter the articles with “studies” (subject > studies) 9.Read the titles and abstracts of your results, select the relevant articles and safe them. 10.No full text available in ABI/Inform or BS Elite? Try Google Advanced.

73. Pit stop EBMgt Learning through play !  Try all buttons  Make lots of mistakes  Have fun !

74. Pit stop EBMgt Assignment: search strategy Search for research articles on teambuilding

75. Pit stop EBMgt 2. Search in ABI/Inform: 1. How many articles has Stephen Covey published in peer reviewed journals? 2. How many of these articles are based on scientific research? 3. Are there articles (by other authors) that are critical of Covey’s 7 Habits? 4. How many of these critical articles are based on scientific research? Exercise: Search for evidence

76. Pit stop EBMgt Evidence-based management Step 3: Critical appraisal of studies

77. Pit stop EBMgt 5-step approach EBMgt is a 5-step approach 1.Formulate an answerable question (PICOC) 2.Search for the best available evidence 3.Critically appraise the quality of the found evidence 4.Integrate the evidence with managerial expertise and organizational concerns and apply 5.Monitor and evaluate the results

78. Pit stop EBMgt Effect vs Non-effect

79. Pit stop EBMgt Internal validity

80. Pit stop EBMgt internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible. Internal validity

81. Pit stop EBMgt Levels of internal validity Levels of internal validity

82. Pit stop EBMgt Which design for which question? Explanation

83. Pit stop EBMgt Best research design?

84. Pit stop EBMgt Critical appraisal

85. Pit stop EBMgt Critical appraisal Postgraduate Course 1.Is the study design appropriate to the stated aims? 2.Was a control group used? 3.Was a pretest used? 4.Are the measurements likely to be valid and reliable? 5.Could bias or confounding have occurred? 6.How large was the effect size?

86. Pit stop EBMgt Levels of internal validity 1.Were there enough subjects in the study? 2.Was a control group used? 3.Were the subjects randomly assigned? 4.Was a pretest used? 5.Was the study started prior to the intervention or event? 6.Was the outcome measured in an objective and reliable way? 6x yes = very high (A) 5x yes = high (A) 4-3x yes = limited (B) 2x yes = low (C) 1-0x yes = very low (D)