1. may 9, 2008 1 Implementation and use of operations research Dr. Lesly Michaud World Vision, Haiti

2. may 9, 2008 2 Presentation Outline Description of implementation Objectives & methods of operations research (OR) Selected OR results Conclusions and implications of OR for overall evaluation

3. may 9, 2008 3 Timeline 2002200320042005 Qualitative/ formative research Development of BCC and 2 program models Implementation of full program models Operations Evaluation (Round 1) Consultation with WV; program improvements Operations Evaluation (Round 2) Baseline Survey Final Survey Development Implementation Impact evaluation

4. Program implementation -Service delivery points -Staffing

5. may 9, 2008 5 Program service delivery points Rally Post Identification of beneficiaries Mandatory attendance at - Mothers Clubs* (mother of beneficiary child) - Home visits (severely malnourished - Prenatal/ postnatal clinics - Mothers Clubs (pregnant/lactating women) - Home visits (post-delivery) Eligible to receive food at Food distribution points *Organized differently in preventive and recuperative PREG/LACTATING WOMEN CHILD BENEFICIARIES

6. may 9, 2008 6 Organizational structure Headquarters (Port-au-Prince) Director of MCHN Director of Commodities Regional Office (Central Plateau) Regional MCH Coordinator; Assistants to MCH Coordinator MCH Supervisors (Nurses) Health Promoters Assistant Health Promoters Food Monitors Regional Office (Central Plateau) Regional Commodities Officer

7. may 9, 2008 7 Services at Rally Posts (RP) Identical services in both program models Services include: Growth monitoring and promotion Determination of eligibility for food assistance* Vaccination Distribution of vitamin A capsules, ORS, deworming medications Brief group education sessions * Based on program model: age-based eligibility in preventive and nutritional status-based eligibility in recuperative

8. may 9, 2008 8 Services at Mothers Clubs (MC) Preventive BCC - Pregnancy: 6 sessions - Lactation: 6 sessions - When child is 6-23 mo: 18 sessions, of which 7 focus on complementary feeding and 11 on other topics; some reinforcement Recuperative BCC - Pregnancy: 6 sessions - Lactation: 6 sessions - Malnourished child is 6-59 mo: 9 sessions of which 7 sessions focus on recuperation, nutrition; 2 on other topics Small group education sessions, using interactive methods, meeting monthly and located close to homes of beneficiary mothers

9. may 9, 2008 9 Services at pre- and post-natal consultations Identical in both program models Services include –Physical examination –Iron-folate supplements –Tetanus toxoid immunization –Post-partum vitamin A

10. may 9, 2008 10 Services at Home Visits Identical in both program models Post-delivery Newborn visit Follow-up of severely malnourished children

11. may 9, 2008 11 Food Distribution Points Monthly ration is the same in both groups (except for longer duration for children in preventive program) Ration for children is different from ration for pregnant/lactating women. Indirect household ration is included, but only one indirect ration even if multiple beneficiaries in a single household

12. may 9, 2008 12 CommodityChildren 6-24 mo or malnourished <5 years Pregnant/lactating women Direct ration (kg) Indirect ration (kg) Direct ration (kg) Indirect ration (kg) WSB8 SFB 1055 Lentils 2.522 Oil2 1.5 WSB=Wheat-soy blend; SFB=Soy-fortified bulgur; oil is fortified with vitamins A & E Rations distributed at FDPs

13. Operations Research

14. may 9, 2008 14 Main objectives OR Round 1 E valuate program implementation relative to plan Identify and prioritize solutions to problems OR Round 2 Evaluate implementation of solutions Look for differences in implementation and/or program use between preventive and recuperative models – To rule out implementation differences as a reason for differences in impact at the final survey

15. may 9, 2008 15 Methods Structured observations Rally Posts, Mothers Clubs, Food Distribution Points Exit interviews with beneficiaries and staff Group interviews with beneficiaries & staff

16. may 9, 2008 16 Examples of issues covered Rally Posts (RP) – OR-1 & OR-2 Staffing and participation Organization of activities, bottlenecks Availability of supplies Large group education: staffing, quality Growth monitoring: accuracy, quality, counseling Mothers Clubs (MC) – OR-1 & OR-2 Organization (age groupings) relative to plan Staff facilitation and teaching skills Food Distribution Points (OR Round 1 only) Organization Quality of food Amounts distributed (compliance with intended amount)

17. may 9, 2008 17 Other issues covered Staff-level factors Motivation, job satisfaction Perceptions of supervision/management Technical knowledge Time allocation/workload Participants Knowledge of BCC messages Trial and adoption of key practices Factors influencing trial and adoption (constraints, facilitating factors)

18. may 9, 2008 18 Key results from Round 1 Rally Posts High participant/staff ratios, crowded Sequence of activities often did not follow plan Problems with availability of supplies Vaccines, ORS, vitamin A, deworming Group education Only one session, crowded, poor ambience, little use of visual materials, did not follow monthly plan for topics Growth monitoring Some errors in weighing and plotting, communication with mothers somewhat limited and vague

19. may 9, 2008 19 Key results from Round 1 Mothers Clubs Overall quality very good Facilitation, teaching, technical content In the preventive model, difficulties to organize MCs by age groups Quality not consistent across all staff/program areas

20. may 9, 2008 20 Key results from Round 1 Food Distribution Points Minor inconsistency in measurement of commodities but beneficiaries largely got intended amount Staff and food transport problems and a very high beneficiary/staff ratio lead to high time burden for beneficiaries and staff Other issues – Food handling/hygiene concerns – Communication issues (health & commodities)

21. may 9, 2008 21 Consultative workshop with staff Port-au-Prince February 2004 Presentation of OR 1 results, dialogue, use of decision tool to: Prioritize problems Identify feasible solutions Make an action plan Program management played an active role in process Field staff were also involved

22. may 9, 2008 22 OR 2 Rally Post Results Organization & supplies 2003 vs 2004 At 10 RPs observed both years: Slight decrease in average participant/staff ratio; range decreased substantially 2003: 21 (7-59) 2004: 17 (6-33) Sequence of activities followed plan designed following workshop Numbering system used to ensure first-come first-served Availability of supplies still a problem: some better, some worse than 2003

23. may 9, 2008 23 OR 2 Rally Post Results Group education 2003 vs 2004 Many positive changes No differences between program areas # of sessions observed 2003 n = 24 2004 n = 62 Mean number of sessions (range) 1.3 (1 - 3) 3.1 (2 – 6) Mean number of participants (range) 30 (4 - 76) 15 (5 – 30) Mean duration of session (range) 10 (1 – 23) 22 (6 – 41)

24. may 9, 2008 24 OR 2 Mothers Clubs Results Large improvement in organizing mothers into groups by child age in preventive MCs Session topics appropriate to groups Quality of facilitation and teaching remains very high (e.g. technical content 84% correct; same as OR1) No differences between program areas

25. may 9, 2008 25 Summary of OR 2 results Staff interviews Staff-level factors Motivation, job satisfaction – very positive Perceptions of supervision – positive, but some variability Technical knowledge – generally very good, some gaps No differences between program areas

26. may 9, 2008 26 Conclusions 1 : Implementation of solutions Achieved success in implementing majority of solutions identified at workshop Demonstrated commitment to program improvement Operations research helped build staff interest and commitment in improving program Areas for continuing improvement were identified

27. may 9, 2008 27 Conclusions 2: Comparing preventive and recuperative program areas No major differences in quality of implementation Some suggestion that communication during growth monitoring is better in preventive areas No major differences in staff-level factors or in participant trial/adoption of practices

28. may 9, 2008 28 Conclusions 3 : Implications for evaluation of effectiveness and cost-effectiveness Any differences in effectiveness can reasonably be attributed to differences in program models, but NOT program implementation