METAEVALUATION An Overview (dls 8/30/11)

Key Questions  1. What is the essence of metaevaluation?  2. Why is metaevaluation important?  3, What are the benefits of sound metaevaluations?  4. What is a basic framework for metaevaluation roles & execution?  5. What are key sources of metaevaluation criteria?  6. What is an operational definition of metaevluation?  7. What are key metaevaluation tasks?  8. What qualifications are needed to conduct metaevaluations?  9. What are some examples of metaevaluation?  10. How is metaevaluation related to meta-analysis?

1. What is the essence of metaevaluation?  Fundamentally, a metaevaluation is an evaluation of an evaluation.  Loosely, a metaevaluation is to an evaluation as a financial audit is to an accounting process and report.  In both cases, the aim is to help assure or to determine that the subject study is sound and that it meets the pertinent professional standards.

2. Why is metaevaluation important?  Evaluation is of fundamental importance to society.  Help assure & confirm cost-effectiveness  Help assure & confirm soundness, safety, & fairness  Help assure & confirm beneficial outcomes  Many things can and do threaten success of evaluations.  Narrow focus  Conflicts of interest  Bias  Unreliable data  Invalid conclusions

3. What are the benefits of metaevaluations?  Help assure but also judge the soundness of evaluations  Assess evaluations for efficiency, effectiveness, & propriety  Provide direction for improving evaluation services  Contribute to professionalizing the evaluation field  Enhances credibility of evaluation services

4. What is a framework for metaevaluation? Internal & External Formative & Summative Metaevaluations Formative MetaevaluationSummative Metaevaluation InternalEvaluator assesses & strengthens the evaluation Evaluator reports an attestation of the evaluation’s adherence to standards for evaluations ExternalIndependent metaevaluator provides feedback for strengthening the ongoing evaluation Independent metaevaluator reports on the completed evaluation’s utility, feasibility, propriety, accuracy, & accountability

5. What are sources of metaevaluation criteria?  AEA (2011). Guiding Principles for Evaluators  GAO (2007). Government Auditing Standards  Joint Committee on Standards for Educational Evaluation (2011). The Program Evaluation Standards

6. What is an operational definition of metaevluation?  Metaevaluation is  the process of delineating, obtaining, & applying descriptive information & judgmental information—about pertinent criteria—including the evaluand’s utility, feasibility, propriety, accuracy, & accountability & its systematic nature, competent execution, integrity, respectfulness, & social responsibility—in order to guide the evaluation & report its strengths & weaknesses.

7. What are key metaevaluation tasks?  1. Staff the evaluation  2. Interact with stakeholdrers  3. Define the metaevaluation questions  4. Agree on standards to judge the evaluation  5. Contract for the metaevaluation  6. Collect & review existing, relevant information  7. Collect new information, as needed  8. Analyze & synthesize findings  9. Judge the evaluation against the chosen standards  10. Report findings  11. Help the client & others to interpret & apply findings

8. What qualifications are needed to conduct metaevaluations?  Working knowledge of evaluation standards  Expertise in qualitative & quantitative methods  Comprehension of the evaluation’s subject program  Successful metaevaluation experience  Training in the standards and tasks of evaluation & metaevaluation  Honesty, integrity, & respect of individuals & society  Skills in contracting  Abilities in communication & collaboration  Ability to staff & manage complex enterprises

9. What are examples of metaevaluation?  Arthur Anderson evaluation of Enron accounting reports  EC’s metaevaluation of the Marine Corps Performance Review System  Metaevaluation of Teach for America’s teacher evaluation system  A metaevaluation of NAGB’s system of levels for judging student performance on NAEP  DLS metaevaluations of state-level evaluations of India’s national primary school reform program  Metaevaluation of and external evaluation of the NYC school district’s Waterford Integrated Learning System

10. How is metaevaluation related to meta-analysis?  A metaevaluation assesses an evaluation.  A meta-analysis quantitatively synthesizes studies that address a common research question, e.g., concerning the differential effects of treatment & control conditions.  Metaevaluations may be used to assess candidate studies for a meta-analysis.  Metaevaluations also can assess a meta-analysis final report.  Meta-analyses are rarely applicable in metaevaluations

Closing  The preceding is a general overview of key matters related to metaevaluation.  Metaevaluation is a special case of evaluation which focuses on evaluation studies, devices, & systems.  Metaevaluation is important to help assure the soundness of evaluations.  Metaevaluation serves evaluation audiences & society by systematically assessing completed evaluations.  Metaevaluations must be keyed to the standards of sound evaluation.  Of course, metaevaluations should themselves be evaluated.