CHAPTER 27: METAEVALUATION For inclusion in Stufflebeam, & Coryn (2011) Evaluation Theory, Models, & Applications 1

METAEVALUATION: THE BASICS Rationale Definition Conceptual Framework Qualifications for Conducting Metaevaluations Metaevaluation in Relation to Meta-Analysis 2

METAEVALUATION: AN ILLUSTRATIVE CASE Teacher evaluation system employed by Teach for America (TFA) TFA recruits, trains, certifies, & places B.A. graduates to teach in inner-city schools TFA can succeed only if its graduates are competent, effective, & credible TFA must rigorously evaluate its students TFA’s evaluation system must be credible 3

AUDIENCES FOR A METAEVALUATION OF TFA’S EVALUATION SYSTEM TFA’s sponsors, administrators, certifying bodies, client school districts State teacher certification bodies The teacher education establishment TFA’s teacher trainees 4

THE METAEVALUATION TEAM Jason Millman—Cornell U. William Mehrens—Michigan State U. Daniel Stufflebeam—WMU (Collectively possessed expertise in evaluation theory & methodology, measurement, statistics, state teacher certification systems, evaluation standards, & metaevaluation methods) 5

TFA’s PERFORMANCE ASSESSMENT SYSTEM (PAS) Teacher-compiled portfolio Portfolio assessors Assessor training & calibration system Portfolio examination Certification recommendations Telephone & face-to-face interviews 6

PORTFOLIO COMPONENTS Teaching plans Video-taped teaching Teacher’s assessment devices Samples of students’ work Analysis of students’ academic growth Survey results from the teacher’s principal, other supervisors, teacher colleagues, parents, & students 7

PORTFOLIO ASSESSESSMENTS Two independent evaluations of each portfolio By specially trained assessors Following specific, standardized rubrics Resulting in sub-scores for certification criteria With a third assessor resolving discrepancies 8

METAEVALUATION JUDGMENTS Performance assessment design & criteria Assessors’ selection & training Portfolio review process Teachers’ impacts on students’ learning Assessors’ ratings of probationary teachers Legal defensibility of PAS Prospects for disseminating uses of PAS PAS’s utility, feasibility, propriety, & accuracy 9

METAEVALUATION PROCEDURES Document analysis (trainees’ academic records, assessors’ credentials, TFA’s plan, recruitment & selection criteria & devices, & training materials) Observation of TFA assessment sessions Secondary analysis of assessors’ ratings Legal analysis (against pertinent laws, statutes, & policies) Ratings against 21 evaluation standards 10

BASIC METAEVALUATION FINDINGS TFA’s evaluators are credible Assessments are legal TFA appropriately reported its assessment results TFA needed to strengthen the PAS, e.g., in less hurried training of assessors, improved assessments of teachers’ effects on student learning, & better matching of assessors & trainees 11

TASKS IN THE METAEVALUATION OF TFA’S PAS 1.Staff the metaevaluation 2.Interact with stakeholders 3.Define questions 4.Agree on standards 5.Contract for the work 6.Collect available info. 7. Collect new information 8. Analyze & synthesize findings 9. Judge the evaluation against standards 10. Report 11. Assist use of findings 12

ILLUSTRATIVE METAEVALUATIONS U.S. Marine Corps Performance Review System N.Y. City School District’s computer-assisted basic skills program Appalachia Lab’s project evaluations Stake & Davis evaluation of Reader Focused Writing prog. Evaluation Plans for India’s primary school reform program U.S. Army’s assessment of course needs in Europe Scriven’s goal-free evaluation of a school improvement project Alternative evaluation approaches Evaluation of Open Learning Australia National Assessment of Educational Progress achievement levels Hawaii’s educational accountability system General Motors system for evaluating executives 13

COMPARATIVE METAEVALUATIONS Assess competing evaluations, evaluation devices, evaluation models, evaluation training programs, etc. Contrast the competitors on metaevaluation criteria—overall and for subsets Examples are seen in Chapters 11 and 27 14

METAEVALUATION CHECKLISTS Program Evaluation Standards Metaevaluation Key Evaluation Checklist Evaluation Contracts Evaluation Budgeting Evaluation Administration Checklist Institutionalizing Evaluation Evaluation Values and Criteria Evaluation Design Plans and Operations Feedback Workshops Reports CIPP Model Deliberative Democratic Qualitative Evaluation Institutionalization of Technology Large-Scale Assessment AEA Guiding Principles Program Evaluation Models 15

CONTEXT & RESOURCE CONSTRAINTS Typically metaevaluation costs are slight—less than 2 % of the subject evaluation’s costs Metaevaluations are “cost-free” if their benefits exceed their costs, e.g., in assessing costly, high-stakes evaluations Nevertheless, not all evaluations need or can afford a metaevaluation, e.g., those who have been judged sound on several previous occasions or very small evaluations 16

EVALUATORS ARE ADVISED TO DEVELOP & EMPLOY AN METAEVALUATION MINDSET Learn, internalize, & regularly apply standards in planning & carrying through evaluations Append to reports attestations of the extent to which the evaluation met a pertinent set of metaevaluation criteria or standards Selectively consult pertinent standards to address problems as they arise 17

WRAP UP re. METAEVALUATIONS Are both formative & summative Serve all segments of society Are useful to both producers and users of evaluations Are applicable to a wide range of evaluations Protect consumers of evaluation reports Are essential for professionalizing evaluation Must be grounded in sound standards 18