1. Improving Accountability by Using External Audit and Verification of Data
Presentation to the Canadian Evaluation
Society/Société Canadienne D’ Évaluation
Vancouver, B.C.
June 4, 2003
Session 302
Kathleen Mooney
SERVE
Assistant Evaluation Specialist
Greensboro, NC

2. External Audit and Verification of Data
What is the strategy?
External Audit and Verification of Data

3. Primary Purpose and Use
Definitions:
Formal review of financial situation or accounts
A methodological examination and review
A third-party evaluation or external evaluation, …an auditor may be a meta-evaluator, typically serving in a formative and summative role
Sources: Merriam-Webster’s Collegiate Dictionary (1998)
Scriven, Micheal. Evaluation Thesaurus (1991)

4. Primary Purpose and Use
Ensure Accuracy
Ensure Reliability
Ensure Utility
Ensure Credibility

5. How Was This Strategy Used in REL Performance Indicators Work?
Strategy Encouraged by Funding Source
Ensured “Buy-in” from all REL Directors
Establish Cost-Sharing from all Labs
RFP/External Bidding Process
Selected External Contactor
Government Auditing Standards
Standards for Evaluating the Quality of Program Performance Data

6. Methodology and Scope Sample Field Visits Telephone Interview
All 10 labs participated in the audit
Field Visits
2 labs
Randomly selected
Aggregating lab/Non-aggregating
Telephone Interview
9 labs

7. Methodology and Scope Data Analysis Flowcharting
Review and Crosschecking
Audit Trail Analysis
Content Analysis
Statistical Analysis
Database Inspection and Accuracy Tests

8. How Was This Strategy Used in REL Performance Indicators Work?
Audit Questions:
How valid (relevant) are the performance indicator data in regard to definitions of the indicators?
How reliable (accurate) are the reported performance indicator data counts and statistics?

9. How Was This Strategy Used in REL Performance Indicators Work?
Audit Questions (cont):
Do the Laboratories have adequate procedures in place to receive, review, verify, process, store, analyze, report and update the performance indicator data?
Are quality assurance controls for these procedures utilized by the Laboratories?

10. Audit Results
“Overall, given that adequate procedures and quality assurance controls where in place, the REL performance indicator data for 2002 were found to be reliable, valid and accurately reported to OERI/IES.”

11. Benefits
Ensures the transparency of the system and data collection methods/results
Improves documentation of processes/results
Provides a seal of approval on data quality
Validated by: “Experienced Public/Private Entity”
Provides recommendations to increase rigor of the data collection system
Survey uniformity
Under-reporting
Intended to increase credibility
of the system

12. Does not always increase credibility Consequences may outweigh rewards
Limitations
Does not always increase credibility
Data collection process will change next year
Consequences may outweigh rewards
Did not sufficiently address the issue of data utility
How can data results be used more effectively at all levels of the system?