1. Chapter 6: Program-Oriented Approaches
Presentation by Jay Kerstetter, Amanda Brown,
Jody Yoos, & Jane Lightner
J.K.

2. Orienting Questions: #1 What are the key concepts of the objectives-oriented evaluation approach? #2 How has this approach influenced evaluation?
What is it?
Objectives-oriented evaluation approach helps determine whether some or all of the program objectives are achieved and, if so, how well they are achieved.
Evaluators may work with stakeholders to establish if program objectives are met.
Information from this approach can assist with deciding to maintain, terminate, or change approaches within the program.
J.K.

3. Tylerian Evaluation Approach
1. Ralph Tyler is credited with initiating this approach in the 1930’s
2. Tyler began to formulate his views on education and evaluation.
3. His approach included the following steps:
Establish goals or objectives
Classify the goals or objectives
Define objectives in behavioral terms
Find situations in which achievement of objectives can be shown
Develop or select measurement techniques
Collect performance data
Compare performance data with behaviorally stated objectives
4. This approach was readily adoptable by evaluators and had great influence on evaluation theorists.
Tylerian Evaluation Approach
J.K.

4. Provus’s Discrepancy Evaluation Model:
Developed by Malcolm Provus; viewed evaluation as a continuous information management process.
Provus stemmed key characteristics of his proposal from Tyler.
Provus viewed evaluation as a process.
This process was called DEM; Discrepancy Evaluation Model, which are broken into four developmental stages.
Definition
Installation
Process
Product
Cost-benefit analysis (optional)
The DEM was designed to facilitate the development of programs in large public school systems and later applied to state evaluations by federal bureau.
The DEM was one of the earliest approaches to evaluation and elements of it can still be found today in many evaluation.
J.K.

5. Orienting Question #3: How is the objective-oriented evaluation approach used today?
Standards-based testing
Accountability in education
Performance monitoring systems used in many government programs
Many refinements to the system since the 1930’s
J.Y.

6. Ralph W. Tyler: Tylerian Evaluation Approach
Influenced the Elementary and Secondary Education Act (ESEA) of 1965, first act to require evaluation of educational programs
Started the National Assessment of Educational Progress (NAEP), only way to examine educational achievement of al 50 states.
J.Y.

7. Malcolm Provus: Provus’s Discrepancy Evaluation Model
Approach evaluated in Pittsburgh public schools
Viewed evaluation as a continuous information- management process designed to serve as “the watch-dog of program-management” and the “handmaiden of administration in the management of program development through sound decision making”
Malcolm Provus: Provus’s Discrepancy Evaluation Model
J.Y.

8. Orienting Question #4a: How are Logic Models used in evaluation?
Developed as an extension of objectives- oriented evaluation
Designed to fill in those steps between the program and its objectives
Program planners/evaluator
Identify program
Inputs
Activities
Outputs (immediate program impacts)
Outcomes (long-term objectives/goals)
J.Y.

9. Today Logic Models are used in program planning/evaluation
Help program staff articulate /discuss how program might achieve goals
What elements are important to evaluate at any given time
Build internal evaluation capacity or think in an evaluative way
Example Companies:
United way of America
WK Kellogg Foundation
Annie E Casey Foundation
J.Y.

10. Orienting Question #4b: How are Program Theories used in evaluation?
Theory-Based evaluation
Is used by evaluators to gain a better understanding of the program
Better define the evaluation questions the study should address
To aid their choices of what concepts to measure and when to measure them
To improve their interpretation of results and their feedback to stakeholders to enhance use.
Orienting Question #4b: How are Program Theories used in evaluation?
J.Y.

11. Steps in Theory Based Evaluation
Engage Relevant stakeholder
Develop a first draft of program theory evaluator or evaluation team
Present draft to stakeholders for further discussion, reaction, and input
Conduct a plausibility check
Communicate findings to key stakeholders
Probe arrows for model specificity
Finalize program impact theory
J.Y.

12. Theory Driven Evaluation
Work with stakeholders to identify key questions to be answered in the evaluation and the appropriate designs and methods for answering those questions
Emphasis is on testing the program model
Provide guidance as to what to measure and when to measure it
The selection of the evaluation questions to be addressed depend on the stage of the program and what stakeholders hope to learn
Provides evaluator with critical information that can be used throughout the evaluation
J.Y.

13. Orienting Question #5: How do theory-based evaluation and objectives approaches differ?
The objectives are identified by group looking for the evaluation.
Purpose of activity specified and then those purposes are being acheived.
Individual credited- Ralph W. Tyler
Theory-Based
The evaluator discusses goals, basics, and objectives of program w/the stakeholders.
The evaluator decides how the program should work and then sees if it works that way.
Huey Chen and Leonard Bickman helped to develop approaches to theory based.
Science based and quantitative.
A.B.

14. Question #6: What are the strengths and limitations of program oriented evaluation approaches?
Objective oriented – the simplicity of the concept and program makes it easy to understand, follow and possibly implement.
Face validity – the evaluator is being held accountable for what is being asked to be evaluated. They want to see if what is currently happening or working based on their own goals and objectives for the program.
A.B.

15. Theory oriented – the chance for the evaluator to partake in dialogue with the stakeholders is a strength of the program because it helps them to expand their knowledge on the topic.
This then leads to the evaluator to have a clear understanding of the program so they know how to properly evaluate the program.
Strengths (con’t)
A.B.

16. Weaknesses:
Objective oriented – the evaluator can have a single minded focus on the objectives which will cause them to overlook the complications, elements and factors contributing to that programs success or failure.
This approach does not ask for the evaluator to gain an understanding of the context in which the program operates this could then be what is affecting the program’s success or failure.
Evaluator could ignore the actual value of the objective. Since the evaluator is told what the objectives are they are not asked to evaluate whether that objective even fits the program.
A.B.

17. Theory oriented – this like the objective oriented can cause the evaluator to ignore certain aspects of the program that are important because the evaluator is so focused on the theory of how it runs and not how it is actually running.
Evaluators may ignore the needs or values of stakeholders involved with the program.
This approach may also oversimplify the complexity of the program making it feel easier to evaluate than it really is because not all of the surrounding factors are accounted for in the process.
Weaknesses (con’t)
A.B.

18. Orienting Question # 7: What is goal-free evaluation?
Rationale: “Goals should not be taken as givens.”
Developed by Scriven (1972)---believes the most important function of a goal-free evaluation is to reduce the bias that occurs from knowing program goals
Thus…increase objectivity in judging the program as a whole.
J.L.

19. Orienting Question #8: What does it teach us about conducting an evaluation?
Goals can act as “blinders” causing us to possibly miss the most important outcomes not related to the goals.
Goal-free evaluation was proposed to primarily indentify the unanticipated side effects that an objectives-oriented evaluation might miss
J.L.

20. Major Characteristics of a Goal-Free Evaluation
1. Evaluator purposely avoids becoming aware of program goals.
2. Predetermined goals are not permitted o narrow the focus of the evaluation study.
3. Goal-free evaluation focuses on actual outcomes rather than intended program outcomes.
4. The goal-free evaluator has minimal contact with the program manager and staff.
5. Goal-free evaluation increases the likelihood that unanticipated side effects will be noted.
J.L.

21. Goal Directed + Goal-Free= can work together
Internal Goal Evaluator vs.
External Goal-Free Evaluator
Goal-directed
How well is program meeting goals
Provide information to administrator
What does the program do?
Looking at ALL the programs outcomes, intended or not
Goal Directed + Goal-Free= can work together
J.L.

22. Information taken from Program Evaluation: Alternate Approaches and Practical Guidelines