Program Evaluation EDL 832 Jeffrey Oescher, Instructor 6 June 2013

Introduction  Objectives Describe the course Understand why evaluation is important Define evaluation Differentiate evaluation and research Discuss the jargon of evaluation

Who Is He?  Jeffrey Oescher Background Current responsibilities Experience as an evaluator Important things

Course Description  Catalogue description Catalogue description  Rationale and goals Rationale and goals  Course organization Course organization  Required text Required text  Course assignments Course assignments  Class schedule Class schedule  Important dates Important dates  Moodle acknowledgment

The Need for Evaluation Why evaluate programs?

The Need for Evaluation  What programs are currently being used in your school? Which ones are working well? Poorly? Are different parts of a program working better than others? What can be done to improve those parts that are not contributing effectively? What are the relative costs and benefits of these programs? Have these programs been planned carefully? Is more planning needed? What adaptations might make the program more effective? Why is a program effective?

Evaluation Defined  OK, so there is a need to evaluate, but what does that term mean?  How would you define the term evaluation?

Evaluation Defined  There are many definitions of this term even among experts in the field.  The identification, clarification, and application of defensible criteria to determine an evaluation object’s value (i.e., worth and merit) in relation to those criteria. Fitzpatrick, Sanders, and Worthen, 2011

Evaluation Defined  Evaluation uses inquiry and judgment to determine standards for judging quality relatively or absolutely; collect relevant information; apply the standards to determine value, quality, utility, effectiveness, or significance; and offer recommendations intended to optimize the evaluation object in relation to its purpose or help stakeholders make decisions.

Evaluation and Research What do you see as the difference(s) between evaluation and research?

Evaluation and Research  Purpose Research  Add knowledge to the field through the development or validation of theory  Establish cause and effect relationships  Seek conclusions Evaluation  Help those who hold a stake in whatever is being evaluated make decisions  Describe aspects of the evaluation object  Seek judgments

Evaluation and Research  Setting the agenda Research  The researcher sets the agenda through the choice of phenomena or hypotheses being investigated Evaluation  The stakeholders set the agenda by determining the focus of the evaluation

Evaluation and Research  Standards used to judge adequacy Research  Validity Internal External  Reliability Evaluation  Accuracy – information is an accurate reflection of reality  Utility – the extent to which the results serve the information needs of the stakeholders  Feasibility – the extent to which the evaluation is realistic, diplomatic, prudent, etc.  Propriety – the extent to which the evaluation is done legally and ethically

Evaluation and Research  Preparation Research  Researcher’s training is focused on methodology within the specific discipline of study Evaluation  Evaluator’s training and expertise is inter- disciplinary

Roles and Activities  Roles Researcher  Conduct the research according to plan Evaluator  Advocate for the use of evaluation  Interact with stakeholders Critical friend to help stakeholders learn Help develop critical judgment Help develop organizational learning Stimulate dialogue among various groups

Roles and Activities  Activities Researcher  Conduct the research (i.e., sample; structure the study; and collect, analyze, and interpret data) Evaluator  Negotiate with stakeholder groups to define the purpose of an evaluation  Develop contracts  Hire and oversee staff  Manage budgets  Work with advisory groups  Collect and analyze data  Communicate with stakeholders  Write reports  Disseminate information

The Jargon of Evaluation  Formative The primary purpose is program improvement  Summative The primary purpose is to provide information to help make decisions about program adoption, continuation, or expansion

The Jargon of Evaluation  Identify a specific situation where you would conduct a formative evaluation? A summative evaluation?  How do you see formative and summative evaluation interacting in the evaluation of a program in your school?  See Figure 1.1, p. 24

The Jargon of Evaluation  Needs assessment Establish whether a problem or need exists and, if so, describe that need or problem as well as make recommendations for ways to reduce the need or correct the problem  Process Describe how a program is delivered  Outcome Describing, exploring, or determining changes that occur in program participants  See Table 1.3, p. 27

The Jargon of Evaluation  Internal evaluations Conducted by program employees  External evaluations Conducted by outsiders  Advantages of either approach See Figure 1.2, p. 31

Using the Jargon  An internal, formative evaluation  An external, formative evaluation  An internal, summative evaluation  An external, summative evaluation

Where to Now?  Course Review the materials provided for you  The text Chapter 1  Review this material in light of our discussion tonight Chapter 2  Read this chapter to appreciate the history of evaluation so you can understand the current context in which evaluators operate Chapter 3  Read this chapter to become familiar with some of the issues that can become problematic when conducting an evaluation  Goal 1 Goal 1  Goal 2 Goal 2