Using indicators to measure country pharmaceutical situations Dr. Edelisa D. Carandang Medicines Policy & Supply Management Department of Technical Cooperation for Essential Drugs and Traditional Medicines (TCM) World Health Organization, Geneva Sept 2006

Using indicators to measure country pharmaceutical situations Concepts on pharmaceutical monitoring and assessment The WHO process on assessing and monitoring pharmaceutical situation Snapshots of current global pharmaceutical situations using Level I and Level II indicator survey- the fact book New approaches in access indicators

Pharmaceutical monitoring and assessment Monitoring Review of the progress toward completing program activities and achieving objectives Allows for corrective action during program implementation Focus on inputs and outputs -integral part of management. Common methods in fully developed monitoring systems Supervisory visits Routine reporting of selected data Sentinel sites Special studies

Pharmaceutical monitoring and assessment Assessment Discussed along with monitoring as part of overall evaluation Analysis of progress and meeting objectives Types of evaluations Needs assessment (situation analysis, Formative evaluation (midterm review) Summative evaluation (final evaluation) Field surveys using standard pharmaceutical assessment indicators & ongoing monitoring system, document review  Strategies for monitoring and assessment developed in parallel for comprehensive unified strategy

Principles in designing monitoring and evaluation system Focus on key monitoring questions and indicators Minimum data collection Develop practical procedures Consider comparisons between facilities and performance over time. Plan how information will be available for timely feedback and follow-up action Phase the design and implementation Design based on the current system

Who can use the results from assessment and monitoring? Countries - focus action, prioritize, measure achievement National policy-makers synchronise policies data and information to donors and other governmental agencies International agencies to assess the structure and capability of countries, assess the progress, accomplishment and impact of aid Professional groups, NGOs and academia to focus advocacy activities and information campaigns Health facilities to be aware of institutional problems & improve situations

Pharmaceutical indicators Variables that measure situations and change (numerical, binomial) Useful tools to track the performance of particular aspects or activities of the pharmaceutical system Linked to an important input, process, or outcome Well-established indicators can be adapted/ modified to reflect the realities Field test

Summarizing indicator measures Percentage: yes or no over total Measures of central tendency Mean: average value, sensitive to outliers, weighed toward skewed value, best summary of normally distributed values Median: middle value, resistant to outliers, good summary of any distribution Equivalent if data are normally distributed Measure of variation 25 th and 75 th percentiles: boundaries of middle half of values, good summary of the overall spread of values, better summary of skewed data

Indicator measure: Ideal/logical values Ideal value 100%-adequate labelling, meds dispensed, adherence to STG, availability of medicines, generic, adequacy of storage 0 days- of stock out Set by countries % affordability Logical value (% use of antibiotics, %injection, ave drug) complex can be calculated empirically 

Indicator measure: group norm Easy for region/facilities to relate to peers Norms may be wrong

Why is it important to use indicators: (1)Setting target Ministry Target = 90%

Why is it important to use indicators: (2) For comparison

Why is it important to use indicators: (3) Seeing trends overtime

Background on WHO work on indicators and pharmaceutical assessment &monitoring Indicator and monitoring tools Indicator for Monitoring National Drug Policy (94 & 99 Ed WHO Operational package for Monitoring & Assessing Country Pharmaceutical Situation Publications 1988 World Drug Situation 2003 World Medicines Situation 2006 Using indicators to measure country pharmaceutical situation: Fact book on WHO Level I and Level II monitoring indicators WHO medicines data base

Develop implementation plans and identify strategies & interventions based on data/information on: availability, affordability, pricing, drug use and regulatory profile, TRIPS, drug management situation. Support implementation of activities and advise in the execution of work plans Indicator-based tools to evaluate structures, processes, outcomes of in countries WHO Evidence-Based Planning and Interventions Guiding Country Works in Medicines WHO Evidence-Based Planning and Interventions Guiding Country Works in Medicines

Why WHO is monitoring and assess pharmaceutical situation using indicators? Objectives prioritized pharmaceutical policy objectives based on evidence determine effects, impact of policy implementation over time establish situation evidence for advocacy Indicators for evidenced based policy Provides evidence for planning, prioritising and identifying interventions Systematic data gathering Allows comparisons & set target Measuring trends 1. Assess and Monitor 2. Plan3. Implement Access & rational use of quality medicines

WHO hierarchical approach to monitoring and assessing pharmaceutical situations Level I Core structure & process indicators Level II Core outcome/impact indicators & household survey Level III Indicator tools for specific components of the pharmaceutical sector ● Pricing●Traditional medicine ● Paediatrics ● Assessing regulatory capacity ● TRIPS ● Supply & Procurement Systematic survey Questionnaire (Health Officials) Level I Questionnaire/rapid assessment/checklist Arrays achievement & weaknessess, illustrate sectoral approaches Level II Comprehensive monitoring of pharmaceutical strategy outcome and impact Measures attainment of objectives Level II More detailed îndicators for monitoring and evaluating specifc areas/components

Level I indicators: structure and process indicators Regular survey questionnaire Advantages Cheap way to get information across countries Can be done repeatedly/regular period Automated questionnaire and data encoding processing Limitations Validation of data Accuracy of responses Missing data

Level II indicators: WHO Operational Package for Monitoring and Assessing county Pharmaceutical Situations Procedure and guidelines Administrative preparation: Coordinating with WHO, ministry/department of health, public health facilities, private drug outlets, warehouses Making logistic arrangements and budget allocations Technical requirements: Selecting geographic areas and facilities Identifying country-specific items of the survey forms, e.g. key basket of medicines, treatment guidelines, etc. Training data collectors to carry out the survey and use the survey and summary forms Analyzing and computing the data Preparing a report and using result

Indicators in the Level II survey 15 survey forms-public health facilities, public pharmacy/dispensary, private pharmacy, warehouses indicators on availability, stock out, record keeping and expiry of key drugs conservation conditions and handling of medicines affordability (child and adult moderate pneumonia and option for other disease condition drug prescribing, dispensing,patient knowledge

Sampling for systematic survey Follow specific procedures to minimize selection bias study population is representative of the reference population A balance between what is desirable and what is feasible- smallest one with a degree of precision

Training data collector for the survey Important for consistency in data gathering process, validity & accuracy of data Who can be trained? Physicians, nurses, pharmacists or paramedical staff Health ministry/department staff and temporary employees (health related background and experience) data collectors from different parts of the country (language differences) Fiji -used students from B.Pharm Y3/Y4,Improve FSm/MoH relations Solomon Islands-p harmacy officers and pharmacists who are not from their own province

Level II Indicator (advantages & limitations) Advantages Practical survey based on small samples Covers impact and outcome in one comprehensive survey Efficient in terms of resource requirement owing to standardized process of training, data gathering and time allocation Limitations Requires more resources for training and conducting systematic survey Limitation in terms number of countries that can be covered

Level III Indicators Systematic survey and monitoring Drug price survey and monitoring Rapid assessment Global survey on Paediatrics medicines Questionnaire on public sector medicines procurement and supply management systems in countries Assessment of regulatory capacity

Fact book on Level I & Level II indicator result Product of years’ of work on developing and improving data-gathering tools, systematic collection of information Overview of pharmaceutical sector components current status & impact Countries grouped low, middle or high-income. Data and information in tables and graphs current situation – 2003 Level I survey progress - comparing Level I 1999 and 2003 surveys impact and outcome measures - Level I & Level II. Country data in CD ROM

Percentage of Countries with Official National Medicines Policy (NMP) in 1999 and 2003

Key pharmaceutical sector legislation Country Income Level LowMiddleHigh Policy area covered: Number % of Countries Number % of Countries Number % of Countries Establishment of regulatory authority Manufacturing of medicines Distribution of medicines Promotion & advertising of medicines Importation of medicines Empowerment to enter premises and collect samples and documentation

Generic prescribing and substitution regulations in 1999 and 2003 Generic Prescribing

Samples Collected for Regulatory Purposes, Tested, and Failing Tests in Previous Year Country Income Level LowMiddleHigh Median [25 th,75 th ] # of samples tested763[172, [433,3382]409[53, 744] (%) of samples that failed test (11%) [3, 108] (3.9%) [4, 106](<1%) [1, 17]

Storage and handling conditions Storage and handling conditions in public pharmacies and warehouses

Countries with insurance that covers medicines Public insurance Private insurance

Medicines Pricing Policies Country Income Level LowMiddleHigh No. % Pricing policy public sector Pricing policy private sector Pricing policy NGO

Proportion of prescribed medicines dispensed % medicines dispensed

Affordability of pneumonia treatment in children

Rational Use of Medicines : Specific Conditions Country Income Level Low (n=12)Middle (n=9) Median % [25 th, 75 th ] Children with diarrhea receiving ORS 80.0 [71.9, 87.5]95.0 [90, 100] (Min. Max) Children with diarrhea receiving antidiarhea 0 [0,40] (Min. Max)5 [0,5] (Min. Max) Children with pneumonia receiving first line antibiotic 90.0 [80, 100]85.0 [70, 100] (Min. Max) Children with pneumonia receiving +1 antibiotic 0 [0,25] (Min. Max)0 [0,0] (Min. Max) ARI patients receiving antibiotic treatment 90.0 [70, 98.8]50 [20, 100] (Min. Max)

Measuring access to essential medicines Level of Access to Essential Medicines Within One Hour Walking Distance* Very low access (<50%) n=18 Low to medium (50%-80%) n=46 Medium to high (81%-95%) n=20 Very high access (>95%) n=20 WHO Region: No. % of Countries No. % of Countries No. % of Countries No. % of Countries Africa Americas East. Mediterranean European South-East Asia Western Pacific *Estimate

Measuring access to essential medicines ( Household Survey) Level I and Level II surveys do not measure access from the patient/consumer perspective. Only household surveys can provide population-based information about how pharmaceutical policies affect the well-being of individuals. Household surveys were tested on a pilot basis With Level II surveys to maximize resources

Measuring access (Composite indicators) Measure policy structures supporting access preconditions) Steps used in the current process Operational Obtain complete data for candidate variables Validate data accuracy Statistical Formal statistical analysis to obtain appropriate factor weights Validate relationship between Level I structure and process and Level II outcomes indicators 4. Reliable systems 5.Affordable prices 3. Sustainable financing 2. ACCESS TO ESSENTIAL MEDICINES 1. Rational selection

National drug policy process Formulation u Identify problems u Define objectives u Develop strategies Monitoring & evaluation u Develop system u Identify tools u Use results Implementation u Develop and execute action plan based on available resources u Prioritize and implement strategies

Medicine Supply Management Management Support Organization Financing Info. Mangnt. HR Selection Distribution Procurement/Use Monitoring Improved drug supply management: Guide selection, procurement, distribution, use

The way forward on country monitoring Evidence through systematic but feasible data collection process is necessary in policy making and activity implementation Should demonstrate that in the long run regular monitoring is not difficult and can be done in a cost efficient manner Portion of country support budget and project grants should be allotted to monitoring and evaluation using indicators Information and data sharing is important

THANK YOU