1. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Research Design for Evaluations Tim Bruckner, PhD, MPH Assistant Professor Public Health & Planning, Policy, and Design University of California, Irvine tim.bruckner@uci.edu Training Programme on Health Workforce Analysis and Planning Rio de Janeiro August 8-13, 2011

2. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Learning Objectives Define the goal of an evaluationDefine the goal of an evaluation Understand internal and external validityUnderstand internal and external validity Identify threats to internal validityIdentify threats to internal validity Be able to describe the following designs: single group case, randomized two-group, non-equivalent two group, and qualitative studyBe able to describe the following designs: single group case, randomized two-group, non-equivalent two group, and qualitative study Describe the strengths and limitations of 3 example study designs Describe the strengths and limitations of 3 example study designs

3. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Definition of Evaluation* l The systematic and objective assessment of an ongoing or completed initiative, its design, implementation and results –Assessments (measurements) are comparable across place & time –Others that repeat the assessment could achieve the same results (repeatable) * Source: Handbook on monitoring and evaluation of human resources for health: with special applications for low- and middle-income countries. Edited by Mario R Dal Poz, Neeru Gupta, Estelle Quain and Agnes LB Soucat. WHO 2009.

4. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Definition of Evaluation* l Especially in HRH evaluations, we must ensure –systematic measurements –measuring of the right variables »Joanne will talk about useful survey designs Source: Handbook on monitoring and evaluation of human resources for health: with special applications for low- and middle-income countries. Edited by Mario R Dal Poz, Neeru Gupta, Estelle Quain and Agnes LB Soucat. WHO 2009.

5. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Research Design and Evaluations l The goal of the research is often to determine whether there exists a causal relation between two variables –Ex: did coordinated clinical care cause better health outcomes? l I will describe research designs often used for human resource evaluations –only a small subset of all designs

6. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. http://www.youtube.com/watch?v=R3zGJdFPBFY&feature=related

7. An Impact Evaluation of Coaches Brazil beats Argentina 3-0 in finals Paraguay beats Brazil in Quarterfinals

8. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. With data only from Copa Americas: Is Dunga a better Coach? What about the team members? Is 2007 team just better? Even if 2007 and 2011 teams had same roster, would aging matter? Are CBF resources to program similar? What about quality of other teams? Have they improved? Were the assistant coaches the real difference makers? So many other potential explanations!!!

9. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Key Goal of Research Design Minimize Threat of Plausible Rival HypothesesMinimize Threat of Plausible Rival Hypotheses use various designs to reduce the plausibility of other explanations Often, a design goal is to use a comparison group that approximates the test group in every way EXCEPT for the variable we want to evaluate

10. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Ideal Research Design use a comparison group that approximates the test group in every way EXCEPT for the variable we want to evaluate In an “ideal world,” to evaluate Dunga vs. Menezes, take the 2007 team  make it the 2011 team and change nothing but the coach This is impossible, but research design attempts to approximate “ideal world”This is impossible, but research design attempts to approximate “ideal world”

11. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Establishing Cause and Effect Temporal precedence Covariation of cause and effect No alternative explanations CauseEffectthen Time if X, then Y if not X, then not Y ProgramOutcome Causes? Alternativecause Alternativecause Alternativecause Alternativecause

12. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Internal Validity Ability to rule out alternative explanations for the findings (stronger validity = fewer alternative explanations) Threats to internal validity History, Maturation, Testing, Instrumentation, Mortality, Regression to the mean Differences in treatment and comparison (control) group before study begins (selection)

13. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. External Validity Extent to which results generalize to a larger populationExtent to which results generalize to a larger population “Generalizability”“Generalizability” ex: Para region as representative of Brazil?ex: Para region as representative of Brazil? ex: survey administered via cell phone rosterex: survey administered via cell phone roster We will focus primarily on internal validityWe will focus primarily on internal validity However, Health Ministers view external validity as crucially important when considering a “scale-up”However, Health Ministers view external validity as crucially important when considering a “scale-up”

14. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Single Group Case AdministerprogramMeasureoutcomesXO Measurebaseline O Alternativeexplanations Alternativeexplanations

15. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Example Example l Increase productivity (visits per hour) – incentives l Increase knowledge of quality procedures - incentives l Pre-post single group design l Measures (O) are visits per hour –Period 0 (beginning) –Period 1 (end)

16. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. History Threat to Internal Validity l Any other event that occurs between pretest and posttest l New wide-spectrum drug available (reduces visit time) –Ex: antidepressants in Sweden ProgramPosttest XO Pretest O

17. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Maturation Threat l Normal growth between pretest and posttest. l High % of new MDs, they get more productive over time anyway. ProgramPosttest XO Pretest O

18. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Testing Threat l The effect on the posttest of taking the pretest l MDs may have learned from the test, not the program ProgramPosttest XO Pretest O l Increase knowledge of quality procedures (incentives)

19. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Instrumentation Threat l Any change in the test from pretest and posttest l Ex: change due to different forms of test, not to program (problem: not systematic) ProgramPosttest XO Pretest O

20. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. “Mortality” Threat l Nonrandom dropout between pretest and posttest l Ex: worst MDs or clinics drop out of program ProgramPosttest XO Pretest O

21. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Regression Threat l Group is a nonrandom subgroup of population (all poor performing clinics) l For example, clinics will appear to improve because of regression to the mean –Previous poor performance was an extreme ProgramPosttest XO Pretest O

22. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Why Does It Happen? Regression artifacts occur whenever you sample asymmetrically from a distribution. Ex: pull the top 10%, or the bottom 10%

23. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Multiple-Group Studies: Threats to Internal Validity

24. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Central Issue l When you move from single to multiple group research the big concern is whether the groups are comparable. l Usually this has to do with how you assign units (for example, persons) to the groups (or select them into groups). l We call this issue selection

25. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Multiple Group Case AdministerprogramMeasureoutcomesMeasurebaseline Alternativeexplanations AlternativeexplanationsXOOOO Do not administer program MeasureoutcomesMeasurebaseline

26. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. ExampleExample l Increase productivity (visits per hour) – incentives l Increase knowledge of quality procedures l Pre-post comparison group design l Measures (O) are visits per hour –Period 0 (beginning) –Period 1 (end)

27. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Selection Threats l Any factor other than the program that leads to posttest differences between groups. XOOOO

28. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Threats to Internal Validity Selection Selection-history Selection- maturation Selection- testing Selection- instrumentation Selection- mortality Selection- regression Social Threats

29. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. “Social” Threats to Validity for Two-Group? All are related to social pressures in the research context, which can lead to posttest differences that are not directly caused by the treatment itself. Most of these can be minimized by isolating the two groups from each other, but this leads to other problems Ex: hard to randomly assign and then isolate

30. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Diffusion or Imitation of Treatment Control providers (i.e., NOT assigned to treatment) might learn about methods to improve productivity from treated people (for example, physician friend in treatment clinic)

31. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Compensatory Equalization of Treatment Administrators give a compensating treatment to controls. Often an attempt to appease upset controls Ex: hints to improve “quality” of procedures =

32. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Compensatory Rivalry Controls compete to keep up with treatment group. May increase productivity as a result

33. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Resentful Demoralization Controls "give up" or get discouraged Out of frustration, they try even less

34. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Types of Designs Random assignment of treatment? Control (comparison) group or multiple measures? YesNo YesNo Randomized or true experiment Quasi-experimentNonexperiment

35. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Two-Group Randomized Experiment

36. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Basic Design RXORXORORORXORXORORO l Proper random assignment assures that we have probabilistic equivalence between groups with respect to all plausible alternative explanations l Differences between groups on posttest indicate a treatment effect

37. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Randomized Experimental Design Kinnersley, P. et al (2000). “Randomised controlled trial of nurse practitioner versus general practitioner care for patients requesting “same day” consultations in primary care.” British Medical Journal 320:1043–8.

38. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Randomized Experimental Design Kinnersley et al. (2000) l Setting: Ten general practices in south Wales and England (near Bristol), 1999 l Research question: what are the differences between care from nurse practitioners and general practitioners for patients seeking “same day” consultations in primary care? l Patients seeking same-day consultations were randomly assigned to a nurse practitioner (NP) or general practitioner (MD).

39. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Kinnersley et al. (2000) l Types of randomization: –Block day or within day (clinic’s choice) l Data:1,368 patients (10 practices) l Outcome measures –Patient satisfaction 2 weeks after initial visit –Resolution of symptoms –Length and quality of consultation

40. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Potential threats to Internal validity –What factors may influence whether a person visits a GP versus NP, and are those factors related to outcome measures? l Patient preferences l Health / co-morbidity of patient l Age of patient l Education of patient l Familiarity with practice

41. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Other issues l External validity –27 NPs invited to participate »12 participated »7 non-respondents and 8 declined l Practical Implementation issue –Randomization was either done by day (4 general practices) or within a day (6 general practices), »By day: disrupts work for general practice; easier to administer »Within day: controls for temporal events related to a day; harder to administer

42. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Findings from Kinnersley et al. (2000) l no difference in reported symptoms 2 weeks after visit –(NP vs GP) l NP spent more time per consultation l On average, increased patient satisfaction with NPs (vs GP) without a significant change in practice or outcomes l However, patients still preferred to see GP next time! l Results support increased use of NPs for “same day” care

43. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Nonequivalent Groups Design

44. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. The Basic Design l Key Feature: Nonequivalent assignment NOXONOONOXONOO

45. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. What Does Nonequivalent Mean? l Assignment is nonrandom. l Researcher did not control assignment. l Groups may be different. l Group differences may affect outcomes.

46. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Barber et al. (2007) l “The Contribution Of Human Resources For Health To The Quality of Care In Indonesia.” Health Affairs 26(3): w367-w379.

47. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Non-equivalent two group design Barber et al. (Health Affairs, 2007) l Setting: Indonesia, 1993-1997 l Research questions: l (1) does eliminating incentive to practice in rural areas reduce rural MDs? l (2) how does the presence and quantity of physicians affect primary health care quality, especially in remote rural areas? l Design: pre- and post-measures –The “X” treatment: beginning in 1992, government froze hiring of civil servants due to decline oil revenues »MDs in rural areas decreased between 1994 and 1998

48. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Two Groups: “Treatment” and control l Rural Java Bali gets the “X” (change in incentive) ’93 ‘97 ’93 ‘97 l Urban Java Bali does not NOXONOONOXONOO

49. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Non-equivalent two group design Barber et al. (2007) l Data: Indonesian Family Life Surveys, 1993 and 1997 (n=7,000 households). Public facility data (n=992 in 1993; n=915 in 1997) l Analytical methods: regression analyses with community-level fixed effects l Measures: –concentration of MDs in facility –Quality of care survey using several case scenarios »Prenatal care, adult care

50. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Internal validity Barber et al. (2007) l Internal validity –What factors other than the reduced incentive may affect where MDs locate? –Are locations related to ability to deliver high- quality care? between 1993 to 1997, MoH developed contracting mechanisms to fill positions between 1993 to 1997, MoH developed contracting mechanisms to fill positions lower technological and facility investment in rural areas may have led MDs to stay in urban areas & reduce quality of carelower technological and facility investment in rural areas may have led MDs to stay in urban areas & reduce quality of care declining social and economic infrastructure of rural communities during the 90’s may reduce retention declining social and economic infrastructure of rural communities during the 90’s may reduce retention

51. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. External validity Barber et al. (2007) l External validity: what population does the sample represent Perhaps study pertains to situation in Indonesia only, for this particular time period, In which elimination of incentive occurred

52. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Barber et al. Findings Health care quality declined everywhere, but large MD reductions in rural outer Java-Bali

53. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Large reduction in MDs, controlling for rivals

54. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. More MDs related to Increase in Quality of Prenatal Care

55. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. What about exploratory studies? l Goal: Hypothesis-generating rather than hypothesis-testing l Qualitative research can l help to understand and categorize the complex environment in which the health worker functions l assist in unravelling the motivation behind certain behaviours l methods typically involve in-depth interviews or observation with key informants

56. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Examples of qualitative inquiry l Why do too few health workers opt for a career in the public sector, favouring private employment instead? l Why do trained nurses choose to remain unemployed ? * Source: Handbook on monitoring and evaluation of human resources for health: with special applications for low- and middle-income countries. Edited by Mario R Dal Poz, Neeru Gupta, Estelle Quain and Agnes LB Soucat. WHO 2009.

57. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. Summary l Most study designs attempt to establish cause and effect relation l Strong designs minimize threats to internal validity l Systematic and objective data collection methods are a core foundation for a research design of HRH evaluation l Qualitative approaches, when rigorously performed, generate hypotheses regarding more complex workforce issues

58. © 2007 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved. And... l I will happily serve as a consultant to evaluate Brazil’s coaching staff in return for World Cup 2014 tickets