Revised OSDV Tools and Processes LFA PSM expert workshop 28-30January 2014

Session objectives 1.Explain why revision was necessary 2.Revised OSDV Templates – What has changed? 3.Clarify changes on the PSM aspects of the OSDV 4.Explain the changes to the OSDV process and requirements 5.Share some field experiences on OSDV implementation

Session structure 1.Presentation - the revised OSDV templates 2.LFA presentation - piloting the revised tool 3.Q&A

Why the revision? Simplify the tool Respond to feedback from LFA Reduce some redundancy - Streamline with other tools Ensure internal consistency - analysis, reporting and interpretation Include provision for Quality of Services verification

What has changed? Modified cover page to provide a snap shot of Data Quality Rating and top Data Quality and PSM issues for Grant Revised M&E and PSM system assessment questionnaires Excel tool scaled down into a worksheet for facilitating the assignment of Data Quality Ratings for indicators Revised the logic for assignment of Data Quality Rating Expanded PSM assessment - included quality of services questions Revised the related tools in OSDV Package

OSDV tool revisions (1) – M&E Assessment Previous VersionArea for ImprovementRevised Version Key findings embedded in text and summarized only at the end of the report Key areas for follow up needed to be more succinct and easily visible Modified cover page to provide a snap shot of DQ ratings per indicator and highlight top DQ and PSM issues Separate form used for identifying indicators and site selection, which was then copied into final report template Required re-entry of data that had been captured in proposal form Required completion and storage of two documents Integrated indicator and site selection into the main report template 1 2

OSDV tool revisions (2) – M&E Assessment Previous VersionArea for ImprovementRevised Version M&E systems were assessed at facility, intermediate and central levels, with raw data provided for each site Central level questionnaire duplicates Implementation Assessment tool Detailed information rarely used for decision-making Simplified M&E system assessment questionnaires. - A single consolidated M&E System Assessment required in OSDV Report – based on LFA analysis and synthesis Excel addendum included detailed information for each indicator and each site assessment Comprehensive data was time-consuming to analyze and not systematically used for decision-making Excel tool scaled down into a worksheet to calculate error margins and assign Data Quality Rating.

OSDV tool revisions (3) – PSM section Previous VersionArea for ImprovementRevised Version OSDV tool included a limited number of PSM- related questions mainly focused on inventory data verification Site visits offer opportunities to collect data on facility-based PSM issues including consumption and patient data Revised and expanded PSM section, with quality of services questions No information available on the alignment of data between facility register and pharmaceuticals & health products data Site visits provide key opportunity to assess the level of coherence between patient and health products data PSM section includes question on the link between facility registers and pharmacy data. e.g.: health products and patient information verification PSM systems assessments and findings were not consolidated Detailed information was not really used for decision-making Require consolidated PSM assessment in OSDV report - based on LFA review analysis and synthesis of findings. 7

OSDV tool revision (4) - Data Quality Rating Data Quality Rating based on Overall Average Weighted Error Margin and the % average report availability for the indicator Overall Good Data Quality: If the Overall Absolute Average Error Margin is less than 10% AND the average % reports availability for the indicator is equal to or more than 90% Minor Data Quality Issues: If the Overall Absolute Average Error Margin is between 10% and 30% AND % average reports availability is equal to or more than 70% Major Data Quality Issues: If the Overall Absolute Average Error Margin is over 30% OR % reports availability is less than 70%

OSDV tool revision (4) - Data Quality Rating Extreme cases with Overall Average Error Margin >30% or <-30%) or overall average availability of reports <70% are flagged as follows: Major– Over reporting: If the Overall Average Error Margin is more than 30% Major – Under reporting: If the Overall Average Error Margin is less than 30% and or % average report availability is less than 70% Note: If the Error Margin is equal to or over 30% at any level, the rating is automatically ‘Major Data Quality Issues’. A rating of ‘Major Data Quality Issues’ also results if the ‘Overall Absolute Error Margin’ is over 30% or the ‘% average report availability’ is less than 70%.

The Revised OSDV templates Report template – Visual display Workbook - Visual display

Revised OSDV Processes & Requirements LFA OSDV Guidelines have been revised OSDV – requirement risk-based In case of OSDV/RSQA - ideally conducted at the same time Excel template to be submitted together with the main OSDV Report Expanded role for LFA PSM experts >> However questions have been designed so that LFA Programmatic/M&E experts are able to collect raw data from facilities.

Expanded Role of LFA PSM Expert At a minimum, the LFA PSM Expert should participate in: -Preparing the OSDV Planning Tool -Briefing Programmatic/M&E Expert on PSM issues -Analysis of data for PSM-relevant sections of the report -Should be involved in reviewing the findings and verifying the summary of issues and making relevant recommendations For some countries (e.g. countries with multiple PSM issues and/or HI Countries CT may require LFA PSM experts to accompany OSDV field team

Revised OSDV Package Revised OSDV package (available online on the Global Fund website) LFA Guidelines for OSDV/RSQA LFA OSDV/RSQA Planning Template OSDV Tool Templates (Report template + Excel Workbook) OSDV User Guide Form uality/

LFA Presentation: Piloting the revised tool

Q&A