may 9, Implementation and use of operations research Dr. Lesly Michaud World Vision, Haiti

may 9, Presentation Outline Description of implementation Objectives & methods of operations research (OR) Selected OR results Conclusions and implications of OR for overall evaluation

may 9, Timeline Qualitative/ formative research Development of BCC and 2 program models Implementation of full program models Operations Evaluation (Round 1) Consultation with WV; program improvements Operations Evaluation (Round 2) Baseline Survey Final Survey Development Implementation Impact evaluation

Program implementation -Service delivery points -Staffing

may 9, Program service delivery points Rally Post Identification of beneficiaries Mandatory attendance at - Mothers Clubs* (mother of beneficiary child) - Home visits (severely malnourished - Prenatal/ postnatal clinics - Mothers Clubs (pregnant/lactating women) - Home visits (post-delivery) Eligible to receive food at Food distribution points *Organized differently in preventive and recuperative PREG/LACTATING WOMEN CHILD BENEFICIARIES

may 9, Organizational structure Headquarters (Port-au-Prince) Director of MCHN Director of Commodities Regional Office (Central Plateau) Regional MCH Coordinator; Assistants to MCH Coordinator MCH Supervisors (Nurses) Health Promoters Assistant Health Promoters Food Monitors Regional Office (Central Plateau) Regional Commodities Officer

may 9, Services at Rally Posts (RP) Identical services in both program models Services include: Growth monitoring and promotion Determination of eligibility for food assistance* Vaccination Distribution of vitamin A capsules, ORS, deworming medications Brief group education sessions * Based on program model: age-based eligibility in preventive and nutritional status-based eligibility in recuperative

may 9, Services at Mothers Clubs (MC) Preventive BCC - Pregnancy: 6 sessions - Lactation: 6 sessions - When child is 6-23 mo: 18 sessions, of which 7 focus on complementary feeding and 11 on other topics; some reinforcement Recuperative BCC - Pregnancy: 6 sessions - Lactation: 6 sessions - Malnourished child is 6-59 mo: 9 sessions of which 7 sessions focus on recuperation, nutrition; 2 on other topics Small group education sessions, using interactive methods, meeting monthly and located close to homes of beneficiary mothers

may 9, Services at pre- and post-natal consultations Identical in both program models Services include –Physical examination –Iron-folate supplements –Tetanus toxoid immunization –Post-partum vitamin A

may 9, Services at Home Visits Identical in both program models Post-delivery Newborn visit Follow-up of severely malnourished children

may 9, Food Distribution Points Monthly ration is the same in both groups (except for longer duration for children in preventive program) Ration for children is different from ration for pregnant/lactating women. Indirect household ration is included, but only one indirect ration even if multiple beneficiaries in a single household

may 9, CommodityChildren 6-24 mo or malnourished <5 years Pregnant/lactating women Direct ration (kg) Indirect ration (kg) Direct ration (kg) Indirect ration (kg) WSB8 SFB 1055 Lentils Oil2 1.5 WSB=Wheat-soy blend; SFB=Soy-fortified bulgur; oil is fortified with vitamins A & E Rations distributed at FDPs

Operations Research

may 9, Main objectives OR Round 1 E valuate program implementation relative to plan Identify and prioritize solutions to problems OR Round 2 Evaluate implementation of solutions Look for differences in implementation and/or program use between preventive and recuperative models – To rule out implementation differences as a reason for differences in impact at the final survey

may 9, Methods Structured observations Rally Posts, Mothers Clubs, Food Distribution Points Exit interviews with beneficiaries and staff Group interviews with beneficiaries & staff

may 9, Examples of issues covered Rally Posts (RP) – OR-1 & OR-2 Staffing and participation Organization of activities, bottlenecks Availability of supplies Large group education: staffing, quality Growth monitoring: accuracy, quality, counseling Mothers Clubs (MC) – OR-1 & OR-2 Organization (age groupings) relative to plan Staff facilitation and teaching skills Food Distribution Points (OR Round 1 only) Organization Quality of food Amounts distributed (compliance with intended amount)

may 9, Other issues covered Staff-level factors Motivation, job satisfaction Perceptions of supervision/management Technical knowledge Time allocation/workload Participants Knowledge of BCC messages Trial and adoption of key practices Factors influencing trial and adoption (constraints, facilitating factors)

may 9, Key results from Round 1 Rally Posts High participant/staff ratios, crowded Sequence of activities often did not follow plan Problems with availability of supplies Vaccines, ORS, vitamin A, deworming Group education Only one session, crowded, poor ambience, little use of visual materials, did not follow monthly plan for topics Growth monitoring Some errors in weighing and plotting, communication with mothers somewhat limited and vague

may 9, Key results from Round 1 Mothers Clubs Overall quality very good Facilitation, teaching, technical content In the preventive model, difficulties to organize MCs by age groups Quality not consistent across all staff/program areas

may 9, Key results from Round 1 Food Distribution Points Minor inconsistency in measurement of commodities but beneficiaries largely got intended amount Staff and food transport problems and a very high beneficiary/staff ratio lead to high time burden for beneficiaries and staff Other issues – Food handling/hygiene concerns – Communication issues (health & commodities)

may 9, Consultative workshop with staff Port-au-Prince February 2004 Presentation of OR 1 results, dialogue, use of decision tool to: Prioritize problems Identify feasible solutions Make an action plan Program management played an active role in process Field staff were also involved

may 9, OR 2 Rally Post Results Organization & supplies 2003 vs 2004 At 10 RPs observed both years: Slight decrease in average participant/staff ratio; range decreased substantially 2003: 21 (7-59) 2004: 17 (6-33) Sequence of activities followed plan designed following workshop Numbering system used to ensure first-come first-served Availability of supplies still a problem: some better, some worse than 2003

may 9, OR 2 Rally Post Results Group education 2003 vs 2004 Many positive changes No differences between program areas # of sessions observed 2003 n = n = 62 Mean number of sessions (range) 1.3 (1 - 3) 3.1 (2 – 6) Mean number of participants (range) 30 (4 - 76) 15 (5 – 30) Mean duration of session (range) 10 (1 – 23) 22 (6 – 41)

may 9, OR 2 Mothers Clubs Results Large improvement in organizing mothers into groups by child age in preventive MCs Session topics appropriate to groups Quality of facilitation and teaching remains very high (e.g. technical content 84% correct; same as OR1) No differences between program areas

may 9, Summary of OR 2 results Staff interviews Staff-level factors Motivation, job satisfaction – very positive Perceptions of supervision – positive, but some variability Technical knowledge – generally very good, some gaps No differences between program areas

may 9, Conclusions 1 : Implementation of solutions Achieved success in implementing majority of solutions identified at workshop Demonstrated commitment to program improvement Operations research helped build staff interest and commitment in improving program Areas for continuing improvement were identified

may 9, Conclusions 2: Comparing preventive and recuperative program areas No major differences in quality of implementation Some suggestion that communication during growth monitoring is better in preventive areas No major differences in staff-level factors or in participant trial/adoption of practices

may 9, Conclusions 3 : Implications for evaluation of effectiveness and cost-effectiveness Any differences in effectiveness can reasonably be attributed to differences in program models, but NOT program implementation