Introduction to Program Evaluation Dr. Suzan Ayers Western Michigan University (courtesy of Dr. Mary Schutten)

Evaluation Defined Systematic investigation of an object’s value (worth or merit) Systematic investigation of an object’s value (worth or merit) Program, project, or instructional material Program, project, or instructional material

Evaluation vs Research Help make judgments/decisions Help make judgments/decisions Stakeholders Stakeholders Specific to the evaluation object Specific to the evaluation object Accuracy, utility, feasibility, propriety Accuracy, utility, feasibility, propriety Interdisciplinary Interdisciplinary Develop knowledge/theory Develop knowledge/theory Researcher Researcher Widespread Widespread Internal validity (causality) & external validity (generalizability) Internal validity (causality) & external validity (generalizability) One discipline One disciplines Purpose: Agenda setter: Generaliz-ability:Criteria:Preparation:

Evaluation Terms Informal: subjective perceptions; absence of breadth, depth; lacking systematic procedures & formally collected evidence; practically useful Informal: subjective perceptions; absence of breadth, depth; lacking systematic procedures & formally collected evidence; practically useful Formal: systematic efforts to define explicit criteria and obtain accurate information Formal: systematic efforts to define explicit criteria and obtain accurate information Evaluation Object: whatever is being evaluated; called evaluand or evaluee throughout WMU Evaluation Center’s website Evaluation Object: whatever is being evaluated; called evaluand or evaluee throughout WMU Evaluation Center’s website

Evaluation’s Purpose Purpose: determine merit or worth of something; render judgments about the value of whatever is being evaluated Purpose: determine merit or worth of something; render judgments about the value of whatever is being evaluated Alternative purposes: serve political functions, learning, social betterment, foster deliberative democracy Alternative purposes: serve political functions, learning, social betterment, foster deliberative democracy

Evaluator’s Roles/Activities Roles: encouraging use, critical friend, facilitator, collaborator, teacher, management consultant, social- change agent, including stakeholders, stimulating dialogue among groups, program planning, scientific expert, provide ‘objective’ opinion Roles: encouraging use, critical friend, facilitator, collaborator, teacher, management consultant, social- change agent, including stakeholders, stimulating dialogue among groups, program planning, scientific expert, provide ‘objective’ opinion Activities: negotiating w/ stakeholders; developing contracts; hiring staff; managing budgets; working w/ advisory panels; collecting, analyzing & interpreting data; writing reports, disseminating information, meeting w/ the press Activities: negotiating w/ stakeholders; developing contracts; hiring staff; managing budgets; working w/ advisory panels; collecting, analyzing & interpreting data; writing reports, disseminating information, meeting w/ the press

Evaluation Process Determine standards for judging quality Determine standards for judging quality Collect relevant information Collect relevant information Apply the standards to determine value, quality, utility, effectiveness or significance Apply the standards to determine value, quality, utility, effectiveness or significance Identify recommendations to optimize evaluation object (program) Identify recommendations to optimize evaluation object (program)

General Uses of Evaluation Program improvement Program improvement Organizational capacity building Organizational capacity building Parent, faculty, staff empowerment Parent, faculty, staff empowerment Information for programmatic decisions Information for programmatic decisions

Uses of Evaluation in Education Empower teachers to have more say about budget allocation Empower teachers to have more say about budget allocation Judge the quality of school curricula in specific content areas Judge the quality of school curricula in specific content areas Accredit schools that meet standards Accredit schools that meet standards Report to an external funding agency on programs it supports Report to an external funding agency on programs it supports

Two Types of Evaluation How the information will be used Formative: developmental feedback focused on program improvement Formative: developmental feedback focused on program improvement Cook tastes the soup Summative: decision making Summative: decision making Guest tastes the soup Figure 1.2 in text (p. 20) Figure 1.2 in text (p. 20) Balancing formative/summative necessary Balancing formative/summative necessary

Relationship Across Program Life EmphasisEmphasis Program life Formative Summative

Process/Outcome Typology (Chen, 1996) Nature of issues/Qs to be examined (Fig. 1.3, p. 22) Needs assessment Needs assessment –Does a problem/need exist? –Recommend ways to reduce the problem Process/Monitoring Process/Monitoring –Description of program delivery Outcome Outcome –Descriptions of changes in recipients or other secondary audiences based on program delivery

Evaluation Sources Figs. 1.4, 1.5 External = evaluations conducted by outsiders External = evaluations conducted by outsiders Internal = evaluations conducted by program employees Internal = evaluations conducted by program employees Pros and cons of each Pros and cons of each Continuum from internal to external Continuum from internal to external

Pros and Cons Knows program well Knows program well Too close to be objective Too close to be objective Full of details but may overlook critical variables Full of details but may overlook critical variables Unbiased Unbiased Difficult to learn as much about program Difficult to learn as much about program Likely to be familiar with important contextual info Likely to be familiar with important contextual info Internal External

Importance and Limitations With so many uses, evaluation is valuable and necessary in our society With so many uses, evaluation is valuable and necessary in our society –Pragamatic, ethical, social, business, intellectual, personal terms Poorly planned evaluations disappoint Poorly planned evaluations disappoint Promising results that cannot be reached create poor feelings and do not yield improvements Promising results that cannot be reached create poor feelings and do not yield improvements

Viewing evaluation as a magic wand to fix all ills often creates disillusionment Viewing evaluation as a magic wand to fix all ills often creates disillusionment Evaluation can identify strengths and/or weaknesses, but it cannot correct problems all by itself Evaluation can identify strengths and/or weaknesses, but it cannot correct problems all by itself Evaluation is a tool for managers and stakeholders to make corrections Evaluation is a tool for managers and stakeholders to make corrections

Recent Trends Influencing Program Evaluation Increased priority/legitimacy of internal evaluation Increased priority/legitimacy of internal evaluation Expanded use of qualitative methods Expanded use of qualitative methods Shift toward using multiple/diverse methods to more fully address evaluation Qs Shift toward using multiple/diverse methods to more fully address evaluation Qs Expansion of theory-driven evaluation Expansion of theory-driven evaluation Increased attention on ethical issues Increased attention on ethical issues Increased use of evaluation in foundations & other non-profit sectors Increased use of evaluation in foundations & other non-profit sectors

Increased education/involvement of stakeholders in conducting evaluations Increased education/involvement of stakeholders in conducting evaluations Increased discussion of the role of evaluators’ advocacy for less powerful stakeholders Increased discussion of the role of evaluators’ advocacy for less powerful stakeholders Technological advances and the related ethical issues raised by such advances Technological advances and the related ethical issues raised by such advances Measuring/Tracking performance by federal gov’t and nonprofit organizations as well as standards- based education Measuring/Tracking performance by federal gov’t and nonprofit organizations as well as standards- based education International growth of evaluation International growth of evaluation