Abu Raihan, MD, MPH Director of Program, Asia IAPB 9th GA, Hyderabad, September 2012 Symposium 6: Challenges in Monitoring Clinical Program Outcomes Developing an Institution-Wide M &E paradigm: The ORBIS experience

2 | ORBIS International Outline Definition and purpose of M&E ORBIS Paradigm: Importance of clinical outcomes Principles of choosing and using key outcomes Multi-tier challenges & barriers to data collection Conclusions

3 | ORBIS International Monitoring Systematic and routine collection of information from the on-going programs for three main purposes: To have internal and external accountability of the resources used and the results obtained To learn from experience to improve practice and take informed decisions To empower partners to deliver quality care

4 | ORBIS International Evaluation Assessing systematically and objectively the value of program. Evaluations help to draw conclusions about five main aspects of the interventions: Relevance Effectiveness Efficiency Impact Sustainability

5 | ORBIS International Key Elements in Designing the M&E Framework Define relevant indicators and their measures; Identify and set benchmarks/standards; Determine data collection strategy; Analyze data and prepare reports for dissemination and decision making.

6 | ORBIS International Purpose of Institution-wide Approach of M&E It is one of the important component of program planning, development and implementation Standardization of data collection and information flow Provides checks and balances over program planning and implementation Strong M&E leads to transparency and good governance for an organization

7 | ORBIS International ORBIS Paradigm: Importance of Measuring Clinical Outcomes To measure excellence in training and practice for quality patient care To provide evidence for advocacy to persuade policy makers, partners to promote good quality eye health Demonstrate impact of good vision on individuals, families and on communities Sustainability ensured when good outcomes attract patients

8 | ORBIS International Principles for Choosing Key Outcomes Outcomes must be appropriate and practical to collect by partners Outcomes reflect quality Outcome data supports Continuous Quality Improvement process

9 | ORBIS International Complexity of Clinical Outcomes There is no one standard for all diseases For certain complex procedures, outcomes are the result of expert consultations –Pediatric cataract, glaucoma –ROP –Retina Current standards are based on international best practices - PPP Current standards are based on international best practices - PPP This is a good place to start, however, these outcomes may need to be adapted as we put them into practice. This is a good place to start, however, these outcomes may need to be adapted as we put them into practice.

10 | ORBIS International Internal Challenges Data collection: who what when how frequent Cost to implement strategy: information system and other resources Which software to manage data on the outcomes? Willingness and ability of staff / partners to collect Analysis, use, and dissemination of results- requires additional skills

11 | ORBIS International External Challenges Cultural issues, not valuing the purpose Suspicion and fear System and resources lacking Privacy / legal issues in some countries to patient data with foreign NGOs Partners feel alienated from M&E process. –Often they have no inputs in the process- “one-size-fits-all” requirements –One-way, upward accountability, very little feedback loop.

12 | ORBIS International Sub-Saharan Africa No systems to collect/collate data No culture to record Suspicion/fear of recording India Limited systems to collate Suspicion/Fear of sharing+/- USA Good patient note Published and publicly available outcome data Lack of information Barriers to data availability Barriers to Outcome Data

13 | ORBIS International Practical Approach to Clinical Outcomes Invest resources for Staff, training, and retention System for data collection and management Develop capacity to analyze data and give partners feedback on the results: close the loop! Build or strengthen capacity of partners toward clinical excellence in practice Bringing Ophthalmologists on board is the key !

14 | ORBIS International Areas for Debate Should current standards be universal across all countries? What are context specific appropriate standards? Ho do we change the culture?

15 | ORBIS International Conclusions Challenges exist, though most can be addressed through appropriate investment in resources Attitudes and perceptions are crucial but are often difficult to change The shift to include measurement of clinical outcomes is critical to good quality eye care programs for elimination of blindness.

16 | ORBIS International Thank you