1 M&E System Workshop on Indicator Set for Rural Water Supply and Sanitation 22 nd November 2007

2 Principles 1.Simplicity. The M&E system should be as simple and robust as possible in the first phase. Once the first phase is successful the system can be made more sophisticated. 2.Improvement rather than replacement. The M&E system should be based as far as possible on improving existing practices rather introducing new ones 3.Flexible and province specific. The M&E system should be flexible, province-centred and adaptable to province specific characteristics. Each province has a different starting point and the transition plan to move from the existing to the proposed system will be different. 4.Cost effectiveness. The M&E system should cost effective and adequately resourced. Provinces can be adopt a sampling approach where appropriate.

3 Principles 5.Accountability. Responsibilities for M&E should be clearly allocated by each province and correspond to mandates of the various decentralised institutions and make use of the best qualified available personnel at local level. 6.Nationally coherent. The centre will ensure coherence by defining the indicators, specifying the minimum provincial reporting outputs, providing provincial support tools for launching and maintaining the M&E system and ensuring timely and coherent national reporting. 7.Independent. Although certain tasks can be outsourced, the database should be under state management and the reporting system should not depend on availability and knowledge of proprietary IT tools 8.Focussed on the end use. Results from the M&E should be made available for planning and decision-making at all relevant levels.

4 Implementation Strategy 10 point Province support package 1.Mandatory minimum specifications – indicator report 2.Mandatory reporting tools 3.Mandatory QA procedures 4.Default data collection system 5.Default templates for data collection 6.Default circulars for officialization 7.Default Training package 8.Budget options and cost norms 9.Guidance on sampling strategy 10.Default Action plan

5 The A-B-C of the Viet Nam RWSS M&E system B.Program level Data related to the particular programs e.g. NPT II Based on the MPI Aligned Monitoring Tool. C.Research level Special studies on sector issues e.g. Value for Money studies; water consumption patterns, gender A.Sector level Sector Indicators

6 A - Sector monitoring A – B – C B –Program Monitoring C – Research /studies NPT II 134 / 135 Donor projects e.g. WB/ ADB E.g. Value for Money Studies Benchmarking/ best practice Human Rights/ Gender studies Functionality Hand-washing Sector coverage indicators 1-8 Program outputs & performance indicators 9-14

7 Sector indicators - A AreaSector level Indicators (national, provincial, district, commune) Water supply1) Proportion of rural population using safe water (%) 2) Proportion of rural population using clean water satisfying standard in decision No.9 issued by MoH Environmental sanitation 3) Proportion of schools having safe water and hygienic latrine (%) 4) Proportion of commune health centers having safe water and hygienic latrine(%) 5) Proportion of public works (Market, cultural house, CPC office) having safe water and hygienic latrine (%) 6) Proportion of HH having hygienic latrine(%) 7) Proportion of livestock farming HH having hygienic farms (%) 8) Proportion of trade village having wastewater and waste treatment system (%)

8 Program indicators - B AreaProgram level Indicators Financial progress9) Proportion of budget implemented compared to budget planned (%) Physical progress10) Number of people served by newly constructed or upgraded facilities per year (#) 11) Number of newly constructed latrines per year (#) Investment efficiency12) Average investment per capital on piped schemes (Vnd/p) Sustainability and service efficiency 13) Proportion of water supply schemes operating sustainably (%) 14) Proportion of piped water supply systems operating under recognized management models (%)

9 Examples of C level surveys AreaPotential indicatorsComment on survey Access  Water & sanitation use  Water quality  Gender Observation and interview to understand e.g. in ethnic minority areas. Detailed water quality Sustainability and Service efficiency  Cost recovery  Institutional models Tariffs, Billing and collection efficiency Expenditure on maintenance. Investment efficiency  Equity  Poverty targeting  Value for money  Private sector  Improvement in Health MOLISSA poverty lists Specialist surveys Behavior Change  Handwashing  Knowledge, Attitude, Practices (KAP) Surveys to test handwashing practice and other hygienic behavior.

10 Indicator: # Variables to be measured:Determination of indicator value: Notes on survey instrument, sampling and frequency of measurement Supportive Definitions: Notes on outstanding issues, options, uncertainties: Indicator specification

11 Indicator 1 water coverage Indicator: 1Proportion of rural population using hygienic water (%) Variables to be measured:Determination of indicator value: 1.Population using hygienic water from dug well (#) 2.Population using hygienic water from tube well (#) 3.Population using hygienic water from tap (#) 4.Population using hygienic water from other water sources, like spring water, rain water, treated surface water, (#) 5.Total population (in commune or sample) (#) Indicator value = variable( )/ variable 5 i.e. The Population using water from one of the safe sources/ total population expressed as a percentage.

12 Indicator 1 continued Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) Frequency of measurement: one a year Supportive Definitions: Safe water has two definitions: Definition i) is that it is water water that satisfies the following quality requirements: No color; No smell; no strange odor; No harmful components that can negatively affect the human health; No harmful bacteria – all by observation. Definition ii) relates to Decision No-09 /2005/QD-BYT issued by MOH. Safe dug well & Safe tube well & Other safe water sources are defined

13 Indicator 1water coverage Notes on outstanding issues, options, uncertainties: The issues to be decided are:  The survey instrument will capture private sector investment as well as public sector investment  The survey instrument is dependent on a correct answer from the householder – it should be considered to introduce a random verification check to improve the assurance of data quality.  Use of HH or individuals as the unit of measurement (better to use HH - more inline with VLSS methodology)  Sample size: A 100% sample would be very expensive. A 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated – GSO might be able to advise given VLSS experience. VLSS sample size is approx 0.07%  Sample selection : a random sample would be best but difficult to guarantee. Possible sources of bias (e.g. near commune center etc) will need to be identified and avoided.  Safe water quality definition i) is used for this indicator. Definition ii) is used for indicator #2  Should the indicator be “access to” rather than “using” – see definition of the sanitation indictor (#6)

14 Indicator 2 water quality Indicator: 2Proportion of rural population using clean water satisfying standard in decision No.9 issued by MoH Variables to be measured: Determination of indicator value: The 22 variables to be measured are defined in the Decision No.9. The water either passes or fails. It passes if all 22 variables are below the maximum limit. It fails if any one variable is above the limit Notes on survey instrument, sampling and frequency of measurement The survey instruments are defined under testing methods in the Decision No.9. Sampling should be 100% but until the budget is available, a random 5% sample size is suggested. Frequency of measurement (6 months for the variables (parameters) 1 to 12 and parameter 21 and 22) the rest depending on the circumstances.

15 Indicator 2 water quality Supportive Definitions: “Safe Water Standard” under decision No-09 /2005/QD-BYT issued by MOH on 11 Mar. ’05 (applied for water supply construction with less than 500 users), and “Drinking Water Supply standard” No-1329 issued by MOH on 18/4/2002 (applied for water supply construction with more than 500 users) should be applied for water quality control in the rural areas. Notes on outstanding issues, options, uncertainties:  Should the indicator by extended to water supplies over 500 users? at present the indicator only covers less than 500 users.  Sample size: A 100% sample would be very expensive – although legally required. For sector information purposes a 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated  Sample selection : a random sample would be best but difficult to guarantee. Possible sources of bias (e.g. near commune center etc) will need to be identified and avoided.  A precise water quality testing of non-piped systems is not measured under this indicator set. Should this indicator be just for piped systems?

16 Indicator 3 schools Indicator 3Proportion of schools having safe water and hygienic latrine (%) Variables to be measured:Determination of indicator value: 1.Presence of safe water & hygienic latrines at primary school point (#) 2.Presence of safe water & hygienic latrines at kindergarten point (#) 3.Total number of primary school points (#) 4.Total number of kindergarten points (#) Indicator value = variable( 1+2)/ variable (3+4) i.e. The number of schools (primary and kindergarten) using water from one of the safe sources and having sufficient latrines/ total number of schools expressed as a percentage

17 Indicator 3 schools Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a school questionnaire which is filled out by a commune appointed official and checked by observation. Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample). Frequency of measurement: one a year Supportive Definitions: Definitions for safe water are given under indicator 1. Definitions for hygienic latrines are given under indicator 6 Notes on outstanding issues, options, uncertainties:  Note: there is a multiple variable involved (both water and sanitation have to be measured at the same school/ kindergarten point)  Should secondary schools be included?  What about satellite school points (especially in mountainous areas)  Sample size: A 100% sample would be very expensive – although legally required. For sector information purposes a 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated  Sample selection : a random sample would be best but difficult to guarantee. Possible sources of bias (e.g. near commune center etc) will need to be identified and avoided.  Not clear what the acceptable rate of water points / hygienic latrines per pupil. A suggestion is sufficient water for at least 5l/pupil/day and not less more 40 pupils per hygienic latrine

18 Indicator 4 health centers Indicator 4Proportion of commune health centers having safe water and hygienic latrine (%) Variables to be measured:Determination of indicator value: 1.Presence of safe water & hygienic latrines at commune health centers (#) 2.Total number of commune health centers (#) Indicator value = variable 1/ variable 2 i.e. The number of commune health centers using water from one of the safe sources and having sufficient latrines/ total number of health communes expressed as a percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a health center questionnaire which is filled out by a commune appointed official and checked by observation. An alternative could be the commune level health authorities to directly provide the information. Sampling: 100% i.e. all commune health centers Frequency of measurement: one a year Supportive Definitions: Definitions for safe water/ hygiene latrines (as indicator 1/6). Notes on outstanding issues, options, uncertainties:  multiple variable involved (both water and sanitation)  Not clear what the acceptable rate of water points / hygienic latrines per commune health center. A suggestion is sufficient water for at least 5l/visitor/day and not less 2 hygienic latrines

19 Indicator 5 – public works Indicator 5Proportion of public works (Market, cultural house, CPC office) having safe water and hygienic latrine (%) Variables to be measured:Determination of indicator value: 1.Presence of safe water & hygienic latrines at commune markets (#) 2.Presence of safe water & hygienic latrines at commune culture house (#) 3.Presence of safe water & hygienic latrines at commune CPC office (#) 4.Total number of commune markets (#) 5.Total number of commune cultural houses (#) 6.Total number of commune CPC offices (#) Indicator value = variable (1+2+3)/ variable (4+5+6) i.e. The number of public facilities using water from one of the safe sources and having sufficient latrines/ total number of public facilities expressed as a percentage

20 Indicator 5 – public works Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a commune questionnaire which is filled out by a commune appointed official and checked by observation. An alternative could be the commune level health authorities to directly provide the information. Sampling: 100% i.e. all commune public buildings of the 3 types specified Frequency of measurement: one a year Supportive Definitions: Definitions for safe water are given under indicator 1. Definitions for hygienic latrines are given under indicator 6 Notes on outstanding issues, options, uncertainties:  Note: there is a multiple variable involved (both water and sanitation have to be measured at the same health point)  Not clear what the acceptable rate of water points / hygienic latrines per commune public facility. A suggestion is sufficient water for at least 5l/visitor/day and not less 2 hygienic latrines

21 Indicator 6 - latrines Indicator 6Proportion of HH having hygienic latrine(%) Variables to be measured:Determination of indicator value: 1.HHs having a hygienic double vault compositing latrine (#) 2.HHs having a hygienic VIP latrine (#) 3.HHs having a hygienic pour flush water seal latrine(#) 4.HHs having a hygienic septic tank latrine (#) 5.Total number of HHs (#) Indicator value = variable( )/ variable 5 i.e. The number of HHs having a hygienic latrine / total number of HHs expressed as a percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official – because of the technical nature of definition, a confirmatory visit should be made. Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) Frequency of measurement: once a year

22 Indicator 6 - latrines Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official – because of the technical nature of definition, a confirmatory visit should be made. Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) Frequency of measurement: once a year Supportive Definitions: The definition of hygiene latrines is given in Decision No.08/2005/QD-BYT of 11/ Notes on outstanding issues, options, uncertainties:  Captures private sector investment as well as public sector investment  Needs correct answer from HH / interviewer– training and random verification check  Sample size: A 100% sample would be very expensive. A 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated – GSO might be able to advise given VLSS experience. VLSS sample size is approx 0.07%  Sample selection : a random sample would be best but difficult to guarantee. Possible sources of bias (e.g. near commune center etc) will need to be identified and avoided.

23 Indicator 7 - livestock Indicator 7 Proportion of livestock farming HH having hygienic farms (%) Variables to be measured:Determination of indicator value: 1. Number of hygienic pig livestock pens 2. Number of hygienic cattle livestock pens 3. Total number of pig & cattle livestock pens Indicator value = variable( 1+2)/ variable 3 i.e. The number of hygienic livestock pens / total number of livestock pens, expressed as % Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a HH questionnaire which is filled out by a commune official Sampling: there is a choice of a 100% sample or if not a random sample Frequency of measurement: one a year Supportive Definitions: “Hygienic livestock pens” – temporarily apply 2 criteria to classify whether the pen is hygienic or not: - At least 10m away from house - The animal excreta volume requires good management to avoid environmental pollution (pens are daily cleaned, excreta volume are covered in storage holes) Notes on outstanding issues, options, uncertainties: Survey instrument largely depend on the proper response from households – need to consider random checking to ensure the data quality.

24 Indicator 8 trade villages Indicator 8Proportion of trade village having waste water and waste treatment system (%) Variables to be measured:Determination of indicator value: 1. Number of villages with solid waste & waste water treatment system 2. Total number of investigated villages Indicator value = variable( 1)/ variable 2 expressed as a percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is a question on a questionnaire on trading village which is filled out by a commune appointed official. Sampling: 100% sampling. Frequency of measurement: once a year Supportive Definitions: “Trading village” has standard definition. The hard definition if adequate waste water waste treatment will be that the discharge standards satisfy the environmental regulations – this requires a sample test or verification that an acceptable technology is in place and adequately maintained and operated. As a softer definition the technical opinion of the commune official will be accepted. Notes on outstanding issues, options, uncertainties: - Either solid waste or waste water is discharged at some certain villages - Available waste water or solid waste system but treatment effectiveness is beyond the requirement.?? - commune officials will need training on what is meant by adequate waste treatment

25 Indicator 9 - budget Indicator 9 Proportion of budget implemented compared to budget planned (%) Variables to be measured:Determination of indicator value: 1.Annual budget planned 2.Annual expenditure Indicator value = variable( 2)/ variable 1 expressed as a percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is simply a recoding of the values in the annual RWSS plan and expenditure records. The AMT should be able to provide this data. Sampling: 100% i.e. all communes and districts and provinces Frequency of measurement: once a year, note expenditure will normally take at least 3 to 6 months after the end of the financial year before it is fully ready. Supportive Definitions: The annual plan is the first PPC approved NTP-RWSS budget and work plan. The expenditure is the audited statements of NTP-RWSS expenditure against plan. Notes on outstanding issues, options, uncertainties:  Does this indicator refer to only capital or also recurrent? if recurrent then it presumably does not include wages which would be very difficult to allocate.  Expenditure of DOH, DARD and the various agencies involved will be needed to be captured – this will not be easy especially for the recurrent budget.

26 Indicator 10 – people served Indicator 10 Number of people served by newly constructed or upgraded facilities per year (#) Variables to be measured:Determination of indicator value: 1.Design capacity in terms of people served for each new scheme 2.Design capacity in terms of people served for major extensions 3.Calculation of ration of actual operational / capacity level /%) for each scheme (phase 2) Indicator value = variable 1 + variable 2 Notes on survey instrument, sampling and frequency of measurement Survey instrument design capacity from the final design documents Phase 2:actual use. Sampling: 100% i.e. all communes and districts and provinces Frequency of measurement: once a year Supportive Definitions: Design capacity in terms of people served : from daily design capacity divided by the assumed daily usage.(need to check the normal design presentations to determine the best way of defining this figure) Calculation of operational level : those who actually receive their main water supply from the newly constructed works Notes on outstanding issues, options, uncertainties:  Distinguish between design capacity and actual service Nos achieved Each year the actual capacity increase will change as more people connect to a network.  Extensions of existing schemes will often need to be estimated and there is a danger of double counting if the extension is done over several years.

27 Indicator 11 latrines constructed Indicator 11Number of newly constructed latrines per year (#) Variables to be measured:Determination of indicator value: No of latrines constructed through subsidies No. of latrines constructed without subsidies (phase 2) Indicator value = variable 1 Notes on survey instrument, sampling and frequency of measurement Survey instrument expenditure records of the number of latrines that were subsidized. A cross check of the controls (visits) undertaken should be made on a spot sample Phase 2: No. of latrines constructed without subsidies should be obtained by calculation / estimate using the difference between variable 1 and changes in year to year in sector indicator # 6. Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year Supportive Definitions: See sector indicator # 6 Notes on outstanding issues, options, uncertainties: It is important in phase 2 to also capture the latrines built not using subsidies.

28 Indicator 12 unit costs Indicator 12Average investment per capital on piped schemes (Vnd/p) Variables to be measured:Determination of indicator value: 1. Total annual investment costs for piped schemes 2. Number of served people from piped schemes (invested in the year) Indicator value = variable 1/variable 2 Expressed in VND/p Notes on survey instrument, sampling and frequency of measurement Survey instrument : recoding of of investment costs for constructed piped schemes and number of served people from those schemes. The AMT should be able to provide this data. Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year. Supportive Definitions: Piped scheme is a water supply system for over 50 households? Notes on outstanding issues, options, uncertainties: - Construction cost price (project) varies due to the market price of materials, equipments, salaries. - Besides water schemes under NTP, there are also schemes funded by other programs/projects. - How to calculate for rehabilitated, expanded schemes (almost like 2 nd investment)?

29 Indicator 13 sustainability Indicator 13Proportion of water supply schemes operating sustainably (%) Variables to be measured:Determination of indicator value: 1. Number of schemes operating sustainably. 2. Total number of current schemes Indicator value = variable 1/variable 2 Expressed in percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is a statistic table to be done by a commune staff based on technical visits. Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year. Supportive Definitions: Water supply systems operating sustainably must meet the following criteria: proper model (this criteria is already measured in the next indicator), properly utilized capacity (≥ 70%), sufficient collection of water tariff, healthy financial mechanism. Notes on outstanding issues, options, uncertainties: - Should the water quality be on criterion in the definition or not? - What if only 3 out of 4 criteria (on criterion is little poor) are satisfied? - Commune staff will need very careful training by pCERWASS

30 Indicator 14 Proportion of piped water supply systems operating under recognized management models (%) Variables to be measured:Determination of indicator value: 1. Number of piped schemes with proper management model 2. Total number of current schemes Indicator value = variable 1/variable 2 Expressed in percentage Notes on survey instrument, sampling and frequency of measurement Survey instrument is a statistic table to be done by a commune staff. Sampling: 100% i.e. all communes and districts and provinces Frequency of measurement: once a year. Supportive Definitions: Piped water schemes under recognized management models include 4 types, namely: - Community management model: - pCERWASS management - - Private management model: - Water supply service business model: Notes on outstanding issues, options, uncertainties: In case new model(s) with effective operation but not recognized, then what ?