1. LIBERIA DATA QUALITY How closely do the data used reflect the truth about results?

2. LIBERIA 2 DATA QUALITY: OBJECTIVES  Understand the 5 Data Quality Standards  Apply a Data Quality Assessment Tool 2

3. LIBERIA 3 Issues MANAGEMENT Can you make decisions based on the data? Better quality data leads to better informed management and planning. REPORTING Are the data believable? Audiences want to know how “credible” your data are so they can trust your analysis and conclusions. 3

4. LIBERIA 4 Five standards for quality of data VALIDITY RELIABILITY PRECISION TIMELINESS INTEGRITY

5. LIBERIA 5 5 Validity Key question: Do data clearly and directly measure what we intend? (7 indicator characteristics?) – Issue: Bias  Result: Modern sanitation practices improved  Indicator: Number of residents in targeted villages who report using “clean household” practices  Source: door-to-door survey conducted three times a year  Most of the people in the targeted region work long hours in the fields during the harvest season Issue: Direct  Result: Poverty of vulnerable communities in conflict region reduced  Indicator: Number of people living in poverty  Source: government statistics office  The government doesn’t include internally displaced people (IDPs) in the poverty statistics

6. LIBERIA 6 6 Reliability Key question: If you repeated the same measurement or collection process, would you get the same data? Issue: Consistency or Repeatability  Result: Employment opportunities for targeted sectors expanded  Indicator: Number of people employed by USAID-assisted enterprises  Source: Structured interviews with USAID-assisted enterprises, as reported by implementing partner AAA, BBB, and CCC  The AO Team found out that the implementing partners were using these definitions: AAA – employees means receives wages from the enterprise BBB – employees means receives full-time wages from the enterprise CCC – employees means works at least 25 hours a week

7. LIBERIA 7 Timeliness Key question: Are data available timely enough to inform management decisions? 7 Issue: How Frequent  Result: Use of modern contraceptives by targeted population increased  Indicator: Number of married women of reproductive age reporting using modern contraceptives (CPR)  Source: DHS Survey  The DHS survey is conducted approximately every 5 years Issue: How Current  Result: Primary school attrition in targeted region reduced  Indicator: Rate of student attrition for years 1 and 2 at targeted schools  Source: Enrollment analysis report from Ministry of Education  In July 2002 the MOE published full enrollment analysis for school year August 2000 – June 2001

8. LIBERIA 8 Precision Key question: Are the data precise enough to inform management decisions? 8 Issue: Enough detail  Result: CSO representation of citizen interests at national level increased  Indicator: Average score of USAID-assisted CSOs on the CSO Advocacy Index  Source: Ratings made by Partner XXX after interviews with each CSO  The SO team reported this data to the Mission Director:  1999 = 2.42 2000 = 3 2001 = 3.000 Issue: Margin of error  Result: Primary school attrition in targeted region reduced  Indicator: Rate of student attrition for years 1 and 2 at targeted schools  Source: Survey conducted by partner. Survey is informal and has a margin of error of +/- 10%  The USAID intervention is expected to cause 5 more students (for every 100) to stay in school longer

9. LIBERIA 9 9 Integrity Key question: Are there mechanisms in place to reduce the possibility that data are manipulated for political or personal gain? Issue: Intentional manipulation  Result: Financial sustainability of targeted CSOs improved  Indicator: Dollars of funding raised from local sources per year  Source: Structured interviews with targeted CSOs  When a AO Team member conducted spot checks with the CSOs, she found out that organizations CCC and GGG counted funds from other donors as part of the “locally raised” funds.

10. LIBERIA 10 Techniques to Assess Data Quality WHY Goal is to ensure AO team is aware of: Data strengths and weaknesses Extent to which data can be trusted when making management decisions and reporting 10 All data reported to Washington must have had a data quality assessment at some time in the three years before submission. ADS 203.3.5.2

11. LIBERIA 11 HOW? Steps to Conduct Assessment 1.Review performance data Examine data collection, maintenance, processing procedures and controls 2.Verify performance data against Agency data quality standards Reliability, precision, timeliness, validity, integrity 3.If data quality limitations are identified, take actions to address them Triangulate; Supplement with data from multiple sources Report the limitations Revise indicator 4.Document the assessment and the limitations in the Performance Indicator Reference Sheet 5.Retain supporting documentation in files Decisions and actions concerning data quality problems Approach for conducting data quality assessment 6.If data will be included in the annual report, disclose the DQA findings in the “data quality limitations” section of the Annual report 11

12. LIBERIA 12 The DQA Tool How organized: follows the life cycle (audit trail) of the data Start with resume: Performance Indicator Reference Sheet. (Skip that part since PIRS was last session) Interview the right people: who knows at each point in the life cycle? Let’s explore the form and try it. Let’s review the protocol. 12

13. LIBERIA 13 Task IX (in pairs) Select a datum from an indicator for which you have data that you know well Pair with another person from a different organization One person interviews the owner of the data Complete the scoring sheet with recommendations in Assessment column Switch roles and repeat 30 mins ea. = 1 hr. 13

14. LIBERIA 14 Session 5. Summary Relation of the 7 indicator characteristics to the 5 data quality standards DQA Tool DQA protocol “There are three kinds of lies: lies, damned lies, and statistics.” --Mark Twain 14