The Quality of Medical Advice: Vignettes (and more) Jishnu Das World Bank

Saving Meena: Story of a death foretold

From the local to the global

Poor Health Life Expectancies (Ethiopia 53; Kenya 54; Zambia 42 in 2007) U5 Mortality (per 1000 born): Ethiopia 127 ; Kenya 114; Zambia 174)

Many countries

Why? Infectious diseases: HIV/AIDS, Malaria, Diarrhea, TB Kenya: TB incidence rate is 62/10,000 (one of the highest in the world) Partly to do with low incomes But see Riley on countries that improved health outcomes at low-income levels Partly to do with access to care Particularly a problem in rural areas

Some quotes There is an obvious difference between rural and urban postings. Working in rural areas involves helping the poor… in urban areas one can learn, have more income, have a good school for one’s children. Doctor in Ethiopia It is Siberia!. Doctor in Addis Ababa There is no plan of development for a doctor in the rural areas; it is as if you are lost. The lack of career development means that it is as if you are punished. Doctor in Rwanda Promotion is as important as remuneration because you cannot stay in the same place forever. Doctor in Ghana Sourced from Serneels and others

And yet… In many countries around the world, people use facilities a lot more than they do in the US

And yet… In deep rural Madhya Pradesh, one of the states in India with the worst HD outcomes, there are 11.3 medical care providers accessible for representative rural households who do not live close to national highways or close to urban centers

And yet… This is not because these villages are huge: for these villages, there are around 20 households per provider

New Studies on Doctor Visits In 5 countries studied, the poor visit doctors almost as much as the rich Kenya: 60% of poorest quintile seek care when sick relative to 78% of richest quintile In urban India, new survey methods show that the poor visit doctors more than the rich The difference arises because of shorter recall periods In rural India, households visit doctors twice as much as in the United States

Do the poor really go to doctors less than the rich? Experimental data from Delhi show that when households are asked on the basis of monthly recall, self-reported doctor visits fall by 65 percent for the poor compared to weekly recall. In fact, the poor go to doctors more than the rich. We don’t know what happens in other countries, because weekly recall questionnaires are very rare!

One alternative Health outcomes may be related to the quality of health care that people receive

Structural Quality Quality traditionally defined as structural quality— state of infrastructure, availability of medicines. This is clearly informative BUT Its not correct when demand is a factor (medicines) The quality of medical advice may be equally (or even more) important The relationship between quality of advice and structural quality is weak (India, Indonesia, Tanzania)

The Quality of Medical Advice: New evidence Since 2002, team working on the quality of medical advice at The World Bank and the University of Maryland Basic Idea What can be measured, and how? How do these measurements help us understand the quality of medical advice? How can the quality of medical advice be improved?

Where we are Quality of medical advice can be decomposed into Competence: What does a doctor or medical care provider know about how to treat an illness Practice: What does a doctor or medical care provider do when faced with an illness We systematically find that the two are very different. This has important policy implications Improving competence is about training Improving practice (given a competence) is about getting doctors to exert greater effort Countries: Tanzania, Urban India, Indonesia, Paraguay Ongoing: Rwanda, Rural India, Argentina

Remainder of Presentation Basic facts about competence Basic facts about practice More interesting facts Lessons learnt (thus far!)

What are Vignettes Standardized mix of (in our case) 5 cases One interviewer is `patient’; other is `recorder and observer’ Child with Diarrhea “My child has been suffering from diarrhea for the last two days, and I do not know what to do” History Doc gets 1 point if he asks about last urination (for instance) Examination Doc gets 1 point if he asks about temperature (IF asked, recorded responds `98.8 F’) Treatment Doc prescribes treatment: treatment is graded by independent raters in South Asia and the US

Scoring Vignettes The questions that doctors ask are compared to a checklist of essential procedures An aggregate “index” is compiled that accounts for the different “difficulties” of different checklist items We call this index “competence” We normalize mean = 0 and standard- deviation=1 Moving from 0 to 2 moves you from 50 th percentile to 95 th percentile

Vignettes: Advantages Advantages Standardized case-load (same 5 cases to all docs) Standardized patient (specify that patient will `comply fully with all medications and tests’

What they know

Public or Private? An MBBS is the formal Indian medical degree—roughly the equivalent of an MD in the US

Distribution of Competence

Regression Models Results very similar in other countries More competent doctors in urban areas More competent doctors in richer areas (within urban and within rural areas) More training increases competence

Competence across countries

Measuring practice Sit in the doctors office Record details about every interaction between the doctor and patient Time History taking Physical exams Drugs prescribed Photo Credit: Ken Leonard

Practice: Some numbers

Another look at practice

Public-Private Again

Table 4: Practice and Competence across Sectors Quintiles of abilityType of providerTotal history questions Probability of. examination (percent)Time spent (minutes)Fees charged (Rs.) LowestPrivate Public nd GroupPrivate Public MedianPrivate Public th GroupPrivate Public HighestPrivate Public Note: This table disaggregates provider practice (observed in the clinic) by competence measured in the vignettes and sector (public/private). History questions refer to the number of questions regarding the illness that the provider asked the patient. An examination consists of any physical contact between the provider and the patient or the use of measuring instruments, such as a thermometer, sphygmomanometer, or stethoscope. Note that an examination only implies that the device was used, not that is was used correctly. Fees charged refers to the total payment at the end of the interaction. Finally, public providers are those who were observed in their public practice and need not be providers who work only in the public sector.

Private MBBSPublic MBBSPrivate, No MBBS What they know, what they do 40% of essential questions asked Percentage of Essential Tasks Completed

Lost Training: Tanzania Sourced from Ken Leonard

Lost Training: India

Training or Effort? Because of lost training We simulate that the impact of training is very small relative to improvements in effort

The big caveat? Is it that public doctors put in a lot less effort because they see many more patients? We find more effort in India in hospitals, which typically see many more patients Two new and incredibly sad pieces of research

Case loads and doctor shortage Maestad and others (2009) went and sat in many doctors clinics in rural Tanzania “The average doctor sees 18.5 patients per day and total time use is 5.7 mins per patient. The doctor completes 22% of essential tasks” That’s less than 2 hours a day in an 8 hour day There is no relationship between caseload and doctor effort!

The Hawthorne Effect Effort jumps when doctors are observed Doctors significantly increase their effort in Tanzania when they know they are being observed with no detrimental effect on patient (Leonard, 2007) Based on Ken Leonard (2007) in Journal of Health Economics

Improving health outcomes New research suggests that improving effort could also improve outcomes Bjorkman and Svensson, 2008: Uganda Rwanda Pay-for-Performance experiment: Ongoing, Rwanda

Summary The quality of medical advice is key and understanding the levels and correlates of quality is an urgent priority Measuring either competence or practice is a good start But measuring both vastly increases our understanding of what is going on and where the policy levers may be Resources exist to help do this A new website ready soon with all studies and resources in one site Support from the Chief Economists office and DEC

Lessons Learnt: What worked (1) The overall methodology is sound and important Sound: Correlates with various characteristics as predicted by common sense Important: Highlights potential and limitations of different policy measures The distribution of quality across public/private or rich/poor The distribution of effort The know-do gap

Lessons Learnt: What worked (2) Initial worry that the variation in doctors is too large in India (Allopathic, Ayurvedas, Unani, Homeopaths) for a single instrument to capture quality Turned out to not be a bit worry because doctors were all treating patients using the same (allopathic) medicines Therefore, they could be graded on the same scale Regardless of the style of the provider’s training, the type of medications dispensed were very similar

Lessons Learnt: What did not work Vignettes We chose the simplest cases possible with no complications Even then, our vignettes are not good at distinguishing bad from very bad doctors (high standard-errors for less than average providers) Perhaps adding in a simple set of written questions would help For instance: “If a child is suffering from diarrhea, what should you give the child?”

Lessons Learnt: What did not work Vignettes All the 5 cases we chose did not require any treatment at the primary level. Therefore, the “mistakes” we usually pick up are errors of commission—doctors doing things that they should not have Definitely consider including cases that require treatment at the primary level (pneumonia?) This allows us to pick up errors of omission

Lessons Learnt: What did not work Practice The vignettes standardize the case-mix and the patient-mix Observing real patients does not. This leads to problems Unobserved patient characteristics could affect inference (for instance sorting) Use an exit-survey to pick some of this up, if possible The cases that overlap with the vignettes are limited (rare cases almost never seen) And may not be perfect overlaps Possibility of using “simulated standardized patients” Pilot currently underway: if this works out, it vastly improves our diagnostic abilities

Lessons Learnt: What did not work Overall The studies thus far are mostly “boutique” studies We are working on how to mainstream them We are not sure what the cost of deviation from the “boutique” approach would be

What I would do (again) Make sure that you keep lots of time for case development Pilot the vignettes until all (>95%) questions that providers ask have a predefined answer Train enumerators until they have memorized the entire vignettes module NEW: use video-recordings of doctor-patient interactions as training material for direct observation (these are being developed)

Things to do differently Post-code treatments (the ultimate nightmare) Make some changes to the direct observation form Patient order Interviewer assessments of patients (to be piloted) Number of questions that patients ask Have a clear idea of the timeline and the work program (but that’s for any of this work!)

Papers on which this presentation is based