Practical Steps to Successful Operational and Implementation Research Jim Foreit The Population Council George Schmid World Health Organization

Doing Operations and Implementation Research

Format Informal Practical—if you want to hear about modeling (a legitimate, important part of OR), run out the door now The session is for you, and us You will forget 90% of what you hear today. So, we've given you materials on four CD-ROMs: –Designing HIV/AIDS Intervention Studies (and other materials) (Pop Council); –Training course materials in operations research (Pop Council) –Framework for Operations and Implementation Research in Health and Disease Control Programmes (and other materials) (Global Fund and WHO) –Other materials (note the FHI guide to qualitative research)

Goals We hope you will remember: –OR is not a mystery –OR need not be difficult –You can—and should—do OR in every programme when you encounter a problem you cannot solve by experience or common sense

What is OR? What is Implementation Research? What is (Public Health) Evaluation? What is Translation Research? OR PHE IR TR

What is OR? What is Implementation Research? What is (Public Health) Evaluation? What is Translation Research? OR PHE IR TR

What is OR? What is Implementation Research? IR OR

Implementation Research "…focuses on how to promote the uptake and successful implementation of evidence-based interventions…that have been identified through systematic reviews." 1 1 Sanders D, Haines A. PLoS Med 2006;3:e186

Implementation Research "…focuses on how to promote the uptake and successful implementation of evidence-based interventions…that have been identified through systematic reviews." 1 1 Sanders D, Haines A. PLoS Med 2006;3:e186 Who does this?

What is OR? What is Implementation Research? IR OR

What is the International Climate Towards OR? The Sydney Declaration (IAS—July 2007) –10% of HIV programme funds for research The Global Fund suggests 5-10% of funds for M&E, including OR Favorable!

Global Fund Grants and "OR and Information Systems" Rounds %52% 2.6%4.4% Korenromp E et al 5 th European Congress on Tropical Medicine and International Health 34% HIV 56% TB 74% Malaria

What is the International Climate Towards OR? The Sydney Declaration (IAS—July 2007) –10% of HIV programme funds for research The Global Fund suggests 5-10% of funds for M&E, including OR PEPFAR February 2008 WHO/World Bank/IAS/GF March 2008 meeting and 4 August Consensus Statement –Extends Sydney Declaration into capacity building and implementation capability Here: IAS session, ANRS session Favorable!

What Skills Do You Need to Do OR? An enquiring mind Effective (epidemiology, statistics, public health, relevant science expertise to the project, e.g., behavioural science, medicine….) Efficient (economics, health services management….) Courses in OR –CDC –Population Council…and….

Format of Session This introduction What is OR? Some practical steps in OR Is it formative or operational research? Your thoughts/discussion Questions!

What Is HIV/AIDS Operations Research? 2008 International AIDS Conference Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc.

Operations Research (OR) is the study of factors under the control of program managers. OR uses research techniques to help choose among alternative uses of resources to meet program objectives.

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. Research Focus Independent variables: manipulated by managers Dependent variables: program outcomes Independent variables: manipulated by managers Dependent variables: program outcomes Factors controlled by managers Outcomes desired by managers and clients

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. Managerial Control Factors Under Control Program systems: training, information, counseling … Factors Not Under Control Cultural beliefs, location, economic status, religion, education…

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. OR Supports Evidence- Based Programs Allows programmes to make evidence-based decisions Identifies service delivery problems Tests service delivery innovations (effectiveness) Tests service delivery costs (efficiency)

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. Costs are a Key Program Factor Study Them ! Cost-effectiveness—what is the least- expensive way to get the desired outcome –e.g., can nurses do the same job as a doctor? Affordability

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. Research Techniques Systematic data collection Qualitative techniques Quantitative techniques Surveys, experiments, focus groups….

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. The OR Process Problem identification Solution generation Solution testing Results dissemination Results utilization

Operations Research for Managers of Reproductive Health Programs © 2008 The Population Council, Inc. OR Requires Collaboration Managers: responsible for decision parameters and desired outcome Researchers: responsible for recommending and implementing research techniques

Taking My Problem Into Operational Research

Q1.What Proportion of Individuals Potentially Helped by Bed Nets are Using Them? 1.2% 2.7% 3.14% 4.24% Morel CM et al. BMJ 2005

Taking My Problem Into Operational Research

First, You Must Recognize a Problem Wanting to know Wanting to act Having the ability to act

Q2. Who Typically Identifies Problems for OR? (more than one answer possible) 1.The patient 2.The programme staff 3.The programme manager 4.The District manager 5.National staff

Zambia Example Problem: TB patients are not being tested for HIV Observation: "TB Corners" where TB patients are seen are in open hallways Intervention: Provide counselling with privacy As Jim said, some issues simply need some common sense, and initiative

Identifying Problems The role of anyone who works in the programme! But, primarily, the role of the manager Identifying problems requires a team approach –Managers must be present in their programmes –Managers must have good relationships with their staff

You've Identified a Potential Problem Patients may not be taking their ART People may not be using condoms Patients may not be returning for follow-up visits Your health care workers seem to have high rates of absenteeism ….

The OR Process Problem identification Solution generation Solution testing Results dissemination Results utilization

There are Lots of Ways to Approach OR Some are simple, others more formal Flexibility needed and desired, to meet the objectives of what you need to do But, let's talk about more formal approaches now….

The Menu of the WHO/GF OR Book 1.Research team 2.Determine Issues 3.Develop proposal 4.Ethical clearance 5.Funding 6.Budget 7.Capacity building 8.Monitor project 9.Pre-test 10.Quality control 11.Stakeholder discussions 12.Dissemination plan 13.Disseminate results 14.Document changes 15.Monitor changes 16.Consider further improvement WHO (TDR)/GF (Jane Kengeya-Kayondo, George Shakarishvili, Serge Xueref) and Bill Brieger and Amy Ellis of Johns Hopkins School of Public Health

Three Phases to OR 1.Planning 2.Implementation 3.Follow Through

Planning 1. Organize the Research Group What disciplines do you need? –Epidemiology, behavioural science…? –Statistics? –Economics (It is difficult to retroactively do the economics part of a study)? Structure Assign tasks, with deadlines Meet regularly Give credit!!!!!!Give credit!!!!!! –Who had the research idea? –Builds teamwork among departments/persons and avoids hard feelings –Pays off in the future

Planning 2. Determine the research questions (and objectives) Be clear, be specific Research question—What do you want to know? –Observation "Our health care workers seem to have high rates of absenteeism" –Research question "What is the rate of absenteeism among our nurses in the Kiev city hospital, and the reasons for it?" Objective –Not: To study adherence –But: To determine rates of adherence among attendees at the XX hospital and reasons for adherence and nonadherence

Planning 3. Develop a research proposal/protocol A document that details: 1. What the research is about and why it is important 2.How the study will be conducted 3.How the results will be used

Planning 3. Develop a research proposal/protocol—"standard parts" Research objectives and questions Background –Literature review –Local context Research team Methods (study design, study population, sampling details, how data will be collected, data management, quality assurance) Data collection instruments Plans for use of data Budget

Planning 3. Develop a research protocol Does every "formal" OR project need a research protocol? –Yes –Makes you clearly state what you will do (and helps to improve your thinking) –Gives you the plan that you will follow, so everyone knows what will be done, and should be done

Q3. Does Every OR Proposal need to go through Ethical Review? 1.Yes 2.No

Planning 4. Obtain ethical clearance

Does Every Study Need to Go Through IRB (Ethics Board)? No…is the project "research," and, does it involve human subjects? What is research? –"A systematic investigation (i.e. gathering and analysis of information) designed to develop or contribute to generalizable knowledge" 1 –"…a class of activities designed to develop or contribute to generalizable knowledge." 2 What are human subjects? –"An individual about whom an investigator… conducting research obtains data through intervention or interaction with the individual or identifiable private information" 1 1 U.S. Office of Research Integrity 2 International Ethical Guidelines Biomedical Research. Council International Organizations Medical Sciences.

Is It Research Involving Human Subjects? Yes No Studying Task Shifting for IMAI Reviewing clinic waiting times records Who makes the determination of whether you need to go to the ethics board? Can use expedited review! Role of the ethics board

Q4. What is the most common thing wrong with consent forms? 1.Do not contain the contact information for the local ethics board 2.Cannot be understood by potential participants 3.Do not mention the word "research" 4.All risks are not revealed 5.Give the patient too much information

Implementation 8. Pre-test all research materials Questionnaires –Is the wording clear? Do people understand what you want? Do the questions ask what they are supposed to? –Test in 5-20 persons Focus groups –Are the guides clear? –Hold one or two focus groups. Consent forms –Do people understand them? Language level? (have your child read it) (sometimes, to check understanding during study, develop a series of 5 or so questions—patient must answer 4 or 5 to enter study)

Q5. Who Need Not Be Part of the Pre-Test Evaluation of Materials? 1.Potential participants 2.Staff who use the materials, e.g., interviewers, focus group leaders 3.Data management staff 4.The members of the research team developing the materials 5.All must be

Implementation 9. Quality Assurance What? During implementation Throughout the study A quality assurance plan should be part of the protocol If you do not perform the study exactly as you said you would, you cannot trust the results

Dissemination 13. Disseminate the results and recommendations If you don't do this, you might as well not have done the study Identify the audiences Create the right documents/strategy/medium for the right audience

Q6. Which Format of Information Sharing Do Decision-makers Like Least (Industrialized Country) 1.Internet/ 2.Meetings/conferences 3.Colleagues 4.Journal articles 5.Short summaries Dobbins M et al. Implementation Sci 2007

Q7. Who Typically Acts on Identified Problems? 1.The patient 2.The programme staff 3.The programme manager 4.The District manager 5.The National staff 6.Too few people act

Four Things Most Commonly Wrong with Proposals 1.The study cannot achieve its objectives 2.Lack of clarity and detail 3.Inconsistency 4.Poor consent forms –Do not contain all needed information –Too complicated and not understandable

Q1.What Proportion of Individuals Potentially Helped by Bed Nets are Using Them? 1.2% 2.7% 3.14% 4.24% Morel CM et al. BMJ 2005

Q2. Who Typically Identifies Problems for OR? (more than one answer possible) 1.The patient 2.The programme staff 3.The programme manager 4.The District manager 5.National staff

Q3. Does Every OR Proposal need to go through Ethical Review? 1.Yes 2.No

Q4. What is the most common thing wrong with consent forms? 1.Do not contain the contact information for the local ethics board 2.Cannot be understood by potential participants 3.Do not mention the word "research" 4.All risks are not revealed 5.Give the patient too much information

Q5. Who Need Not Be Part of the Pre-Test Evaluation of Materials? 1.Potential participants 2.Staff who use the materials, e.g., interviewers, focus group leaders 3.Data management staff 4.The members of the research team developing the materials 5.All must be

Q6. Which Format of Information Sharing Do Decision-makers Like Least (Industrialized Country) 1.Internet/ 2.Meetings/conferences 3.Colleagues 4.Journal articles 5.Short summaries Dobbins M et al. Implementation Sci 2007

Q7. Who Typically Acts on Identified Problems? 1.The patient 2.The programme staff 3.The programme manager 4.The District manager 5.The National staff 6.Too few people act

Discover if you are a “natural” at doing Operational Research and win a valuable prize! International AIDS Conference 2008

Operational Research Operational Research improves programs by testing new service delivery approaches to improve program outcomes Service delivery approach = factor under control of manager Program outcomes = important success indicators OR research question: Will having CHWs visit the homes of newly delivered mothers with HIV+ infants result in more Neviripine treatment within 72 hours?

Formative Research Formative Research identifies health or program problems that need correction. Unlike operations research it does not test solutions to the problem Health problem = The HIV prevalence rate is 5% Program problem = HIV prevalence is three times as high in province X than in province Y, but we have 90% of our program resources in province Y! For all you health workers: Formative research diagnosis program problems, operations research treats program problems

Operational? Formative? Neither? Examples Title: “ Diversification of HIV-1 groups M and O” Before answering: Is this a program problem? Can we use the study to identify a program problem Can we use the study to solve a program problem?

Operational? Formative? Neither? Title 1: “ A comparison of the effectiveness of two approaches in recruiting men for adult circumcision” (Question 8) Title 2: “Differences in the quality of treatment provided MSM and other AIDS patients in an Indian hospital” (Question 9)

Operational? Formative? Neither? Title 3: “HIV prevalence rates among the elderly in rural Illinois” (Question 10) Title 4: “The impact of three types of supervision on frequency of client counseling and testing by clinic workers” (Question 11)

Operational? Formative? Neither? Title 5: “Survey of acceptability of male circumcision among mothers in Mysore India” (Question 12) Title 6: “Efficacy of an HIV prevention curriculum in South Africa: Results of a pilot study” (Question 13)