1. Postgraduate Course 6. Evidence based management: What is the best available evidence?

2. Postgraduate Course 5-step approach EBMgt is a 5-step approach 1.Formulate an answerable question (PICOC) 2.Search for the best available evidence 3.Critically appraise the quality of the found evidence 4.Integrate the evidence with managerial expertise and organizational concerns and apply 5.Monitor and evaluate the results

3. Postgraduate Course Why are disciplines as such as psychoanalysis, astrology and parapsychology widely regarded as pseudo-science? Intermezzo

4. Postgraduate Course “It is easy to obtain evidence in favor of virtually any theory, but such ‘corroboration’ should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction, which might conceivably have been false. … A theory is scientific only if it is refutable by a conceivable event. Every genuine test of a scientific theory, then, is logically an of a scientific theory, then, is logically an attempt to refute or to falsify it.” Falsifiability Carl Popper

5. Postgraduate Course “Inspect every piece of pseudoscience and you will find a security blanket, a thumb to suck, a skirt to hold. What have we to offer in exchange? Uncertainty! Insecurity!” Falsifiability Isaac Asimov

6. Postgraduate Course Research designs What is the BEST car?

7. Postgraduate Course Which design for which question? Research designs

8. Postgraduate Course Explanation Which design for which question?

9. Postgraduate Course Best research design?

10. Postgraduate Course Best available?

11. Postgraduate Course The best available evidence = Studies with the highest internal validity Studies with the highest external validity

12. Postgraduate Course 1. Best available evidence: internal validity

13. Postgraduate Course internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible (confounding). Internal validity

15. Postgraduate Course Three criteria: 1.the "cause" and the "effect" are related 2.the "cause" precedes the "effect" in time 3.there are no plausible alternative explanations for the observed effect When do we know there is causal relation? Causality

16. Postgraduate Course Considerations for research: Causality 1.Are the "cause" and the "effect” related: effect size 1.Does the "cause" precedes the "effect" in time: before and after measurement 2.Are there no plausible alternative explanations for the observed effect: randomization, control group INTERNAL VALIDITY

17. Postgraduate Course internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible ( confounding ). Internal validity

18. Postgraduate Course  Bias  Confounding  Reverse Causation Methodological pitfalls

19. Postgraduate Course Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted. NB: If bias is not taken into account then any conclusions drawn may be wrong! Bias

20. Postgraduate Course 1. Selection bias 2. Information (detection) bias 3. Performance bias 4. Exclusion (attrition) bias 5. Publication bias …… 30. ….. Forms of bias

21. Postgraduate Course Error in the way participants in a study were selected. Because of this comparison groups differ in measured or unmeasured baseline characteristics. Selection bias

22. Postgraduate Course Distortion of the outcome due to misinterpretation of information or systematic errors in the the measurement of research variables which leads to misclassification. Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors. Types of information bias:  Reporting bias (recall bias)  Observer bias (interviewer bias, halo-effect) Information bias

23. Postgraduate Course Confounding is the idea that a 3rd variable can distort or confuse (or confound..) a relationship between two other variables. For instance, when factor X causes disease Y, that relationship could be confounded by factor C that is associated with both factor X and disease Y. C would be an alternative explanation for the relationship observed between X and Y. Confounding

24. Postgraduate Course What are the confounders? 1.Shoe size & quality of handwriting 2.Body length & body weight 3.Number of storks & birth rate 4.Smoking youngsters & better lung function

25. Postgraduate Course Confounding

26. Correlation does not equal causation!

27. Postgraduate Course http://kill-or-cure.heroku.com/

28. Postgraduate Course Reverse causation

29. Postgraduate Course ? Successful companies Charismatic leaders Reverse causation

30. Postgraduate Course Cause and effect can be established only through the proper research design: no amount of statistical hand waving can turn correlations into conclusions about causation !!! Internal validity

31. Postgraduate Course Levels of internal validity Levels of internal validity

32. Postgraduate Course Levels of internal validity Levels of internal validity It is shown that … It is likely that … Experts are of the opinion that … There are signs that …

33. Postgraduate Course The levels of internal validity can only be used to determine which type of research is the best method to assess the validity of the cause-and-effect relationship that might exist between an intervention (or moderator) and its outcomes. In this respect, cross-sectional studies and case-studies have the ‘weakest’ design. This of course doesn’t mean that cross-sectional studies and case-studies have a weak design overall. After all, different types of research questions require different types of research designs. A case study for instance is clearly a strong design for assessing why or in which way an effect has occurred, but obviously not the most suitable design for assessing the strength of a possible cause-and-effect relationship. Keep in mind!

34. Postgraduate Course But … sometimes observational studies are as good as RCT’s Internal validity When the size of effect is very large (swamps the bias)

35. Postgraduate Course These treatments have not been tested in RCTs: are they supported by poor evidence? Internal validity Heimlich manoeuvre Dehydration: drinking water Cardiac arrest: AED

36. Postgraduate Course 2. Best available evidence: external validity

37. Postgraduate Course Ecological validity: Is your organization so different from those in the study that its results may be difficult to apply? Population validity: Is your population so different from those in the study that its results may be difficult to apply? External validity: generalizability Always ask yourself to what extent the evidence is generalizable to your situation:

38. Postgraduate Course Generalizability Generalizability Same Population? Same Intervention? Same Comparison? Same Outcome? Same Context?

39. Postgraduate Course Generalizability Generalizability Keep in mind: What works in one narrowly defined setting might not work in another, but some psychological principles are generalizable to all human beings. but some psychological principles are generalizable to all human beings.

40. Postgraduate Course Internal vs external validity Internal vs external validity All research designs are flawed – though each is flawed differently. For instance, research designs with a high internal validity, such as controlled studies, may be less suited to generalization, which restricts their practical usability. Sample surveys and field research, on the other hand, have lower internal validity, but can sometimes be more useful for management practice. So there is always a trade off between internal validity (precision in control and measurements of variables) and external validity (generalizability with respect to populations, setting and context).

41. Postgraduate Course Best available evidence? Best available evidence? internal validity often high internal validity often low external validity often low external validity often …? external validity sometimes high