1. Introduction to Program Evaluation Dr. Suzan Ayers Western Michigan University (courtesy of Dr. Mary Schutten)

2. Evaluation Defined Systematic investigation of the worth or merit of an object; e.g., a program, project, or instructional material Systematic investigation of the worth or merit of an object; e.g., a program, project, or instructional material (JCSEE, 1994)

3. Evaluation Terms Informal = subjective perceptions Informal = subjective perceptions Formal = systematic efforts to define explicit criteria and obtain accurate information Formal = systematic efforts to define explicit criteria and obtain accurate information Evaluation Object = whatever is being evaluated Evaluation Object = whatever is being evaluated

4. Evaluation Process Determine standards for judging quality Determine standards for judging quality Collect relevant information Collect relevant information Apply the standards to determine value, quality, utility, effectiveness or significance Apply the standards to determine value, quality, utility, effectiveness or significance Identify recommendations to optimize evaluation object (program) Identify recommendations to optimize evaluation object (program)

5. General Uses of Evaluation Program improvement Program improvement Organizational capacity building Organizational capacity building Parent, faculty, staff empowerment Parent, faculty, staff empowerment Information for programmatic decisions Information for programmatic decisions

6. Uses of Evaluation in Education Empower teachers to have more say about budget allocation Empower teachers to have more say about budget allocation Judge the quality of school curricula in specific content areas Judge the quality of school curricula in specific content areas Accredit schools that meet standards Accredit schools that meet standards Report to an external funding agency on programs it supports Report to an external funding agency on programs it supports

7. Historical Overview 2000 BC: formal evaluation by Chinese who gave civil service exams to measure proficiency 2000 BC: formal evaluation by Chinese who gave civil service exams to measure proficiency Socrates used verbal evaluations in education Socrates used verbal evaluations in education 1700’s: “state-istics” were tabulated for mortality, health, and population 1700’s: “state-istics” were tabulated for mortality, health, and population

8. 1800’s: Great Britain & US began evaluating educational and social institutions 1800’s: Great Britain & US began evaluating educational and social institutions Early 1900’s: Educational testing began in earnest (norm-referenced) Early 1900’s: Educational testing began in earnest (norm-referenced) 1950’s, Tyler: planted the seeds of criterion-referenced as an alternative to norm-referenced tests 1950’s, Tyler: planted the seeds of criterion-referenced as an alternative to norm-referenced tests

9. WWII researchers investigated gov’t programs for military in areas of morale, resistance to propaganda, training WWII researchers investigated gov’t programs for military in areas of morale, resistance to propaganda, training This marked the beginning of program evaluation This marked the beginning of program evaluation Kennedy/Johnson era of social programs led to creation of PPBS Kennedy/Johnson era of social programs led to creation of PPBS Planning, programming, and budgeting system of McNamara was applied to assess social programming effectiveness Planning, programming, and budgeting system of McNamara was applied to assess social programming effectiveness

10. Evaluation Requirements in 1965 Education Law ESEA: Elementary and Secondary Education Act of 1965 ESEA: Elementary and Secondary Education Act of 1965 Provided huge increase in federal grants to local schools Provided huge increase in federal grants to local schools Required filing an evaluation report showing funding results, first time educators were required to evaluate their own efforts Required filing an evaluation report showing funding results, first time educators were required to evaluate their own efforts

11. Why evaluate? Evaluation directs actions Evaluation directs actions Identifies needs Identifies needs Aids in monitoring/adjusting programs Aids in monitoring/adjusting programs Yields quality information for intelligent decision-making Yields quality information for intelligent decision-making

12. Two Types of Evaluation Formative: developmental feedback Formative: developmental feedback Summative: decision making Summative: decision making

13. Relationship across program life EmphasisEmphasis Program life Formative Summative

14. Evaluation Sources External = evaluations conducted by outsiders External = evaluations conducted by outsiders Internal = evaluations conducted by program employees Internal = evaluations conducted by program employees Pros and cons of each Pros and cons of each

15. Pros and Cons Knows program well Knows program well Too close to be objective Too close to be objective Full of details but may overlook critical variables Full of details but may overlook critical variables Unbiased Unbiased Difficult to learn as much about program Difficult to learn as much about program Likely to be familiar with important contextual info Likely to be familiar with important contextual info Internal External

16. Importance and Limitations With so many uses, evaluation is valuable and necessary in our society With so many uses, evaluation is valuable and necessary in our society Poorly planned evals disappoint Poorly planned evals disappoint Promising results that cannot be reached create poor feelings and do not yield improvements Promising results that cannot be reached create poor feelings and do not yield improvements

17. Viewing evaluation as a magic wand to fix all ills often creates disillusionment Viewing evaluation as a magic wand to fix all ills often creates disillusionment Evaluation can identify strengths and/or weaknesses, but it cannot correct problems all by itself Evaluation can identify strengths and/or weaknesses, but it cannot correct problems all by itself Evaluation is a tool for managers and stakeholders to make corrections Evaluation is a tool for managers and stakeholders to make corrections

18. Recent Trends in Evaluation Specialists trained in evaluation Specialists trained in evaluation Development of unique evaluation content Development of unique evaluation content Development of formal evaluator preparation programs Development of formal evaluator preparation programs Career opportunities in evaluation emerge Career opportunities in evaluation emerge

19. Institutionalization of the function of evaluation Institutionalization of the function of evaluation Procedures for certification of evaluators Procedures for certification of evaluators Development of professional associations Development of professional associations (JCSEE: http://www.wmich.edu/evalctr/jc/ ) http://www.wmich.edu/evalctr/jc/ Criteria for membership in associations are established Criteria for membership in associations are established

20. Influence of evaluation associations on evaluator preparation programs Influence of evaluation associations on evaluator preparation programs Development of standards for evaluation practice Development of standards for evaluation practice http://www.wmich.edu/evalctr/jc/