THIS PRESENTATION IS INTENDED AS ONE COMPLETE PRESENTATION. HOWEVER, IT IS DIVIDED INTO 3 PARTS IN ORDER TO FACILITATE EASIER DOWNLOADING AND VIEWING, THEY ARE LABELED WEEK ONE PART 1, WEEK ONE PART 2 AND WEEK ONE PART3
Why should I learn about P.E. You will either have to do one or You will participate in one Most people in your agency will not know what a good P.E. looks like Somebody needs to know how to find out if a program is ACTUALLY working Knowing how to conduct a PE will help you know what a good program looks like Many of you will be supervisors of programs It will help you know what to look for in order to improve your program It is a valuable skill that can lead to advancement It is the next level of knowing whether your clients are getting better and if the improvement is due to the program Funding is often dependent on demonstrating program effectiveness
What is program evaluation? Systematic collection of information about the activities characteristics, and outcomes of programs, for use by specific people to reduce uncertainties, improve effectiveness and/or efficiency, and make decisions with regard to what those programs are doing and affecting. Most evaluations are aimed at assessing the program goal attainment!!!!! How effective (does the program work?) or how efficient (how does the program work?) is the program in meeting measurable goals and/or objectives?
A PROPOSAL FOR A PROGRAM EVALUATION ADDRESSES ALL OF THESE: 1. IS THERE A PROGRAM? HOW DO YOU KNOW? WHAT IS IT? 2. THE PURPOSE OF THE EVALUATION SHOULD BE CLEAR AND CLEARLY STATED!!!!!!!!! THIS MEANS STATING BEFOREHAND ‘HOW IT WILL BE USED, WHO THE STAKEHOLDERS ARE AND WHAT THE STAKEHOLDERS WANT TO KNOW!!!! THE RESEARCH QUESTIONS 3, IS IT ACTUALLY FEASIBLE TO DO AN EVALUATION NOW? 4, CONCRETE EVALUATIVE CRITERIA SHOULD BE AGREED UPON AND ESTABLISHED BEFORE THE EVALUATION IS CONDUCTED!!!!!!!! THAT IS, YOU SHOULD KNOW BEFORE YOU UNDERTAKE THE EVALUATION, WHAT SPECIFIC THINGS YOU ARE GOING TO EVALUATE AND HOW THESE ‘THINGS’ TO BE EVALUATED RELATE TO THE PROGRAM. YOU MUST HAVE SOURCES OF DATA. YOU MUST KNOW WHERE THE DATA YOU USE TO EVALUATE THE PROGRAM IS LOCATED OR HOW IT CAN BE CREATED! 5. THE MEASUREMENT PROCEDURES AND INSTRUMENTS SHOULD BE APPROPRIATE TO THE GOALS AND EVALUATIVE CRITERIA. 6. ARE THERE THINGS THE STAKEHOLDERS SHOULD KNOW ABOUT THE CONSEQUENCES OF AN EVALUATION BEFORE IT IS CARRIED OUT. SOMETIMES IT BECOMES CLEAR THAT CARRYING OUT AN EVALUATION AT THE CURRENT TIME WOULD PROVE DISASTROUS!!!
But first you must know if there is an actual program. QUESTION ONE: WHAT IS THE PROGRAM TO BE EVALUATED What is a program? PROGRAMS ARE DESIGNED TO ADDRESS SPECIFIC PROBLEMS FOR A PARTICULAR POPULATION. EXAMPLES a program for pregnant teens whose purpose is to help them a. be responsible parents and b. keep them in school. a program to reduce pregnancy in high school a program designed to reduce truancy in highschool a program designed to help people on welfare get viable employment a program designed to increase independent living skills in people with chronic and severe psychiatric diagnoses. a program designed to place and maintain people with chronic and severe diagnoses in independent living quarters a summer day program designed to reduce hospitalization in adolescents who are at risk for psychiatric hospitalization or jail, during the summer, when school is out. Notice how each program identifies 1. the problem. 2. the population to be served. And 3. its focused purpose!
What is a program It is not an agency, an organization or services. A program is a sub-system of an agency or a self-contained system that has staff, clients, services, data/records, a budget and resources which are designed to achieve specific objectives, or the goal(s) of an organization; or specific objectives derived from a mission statement. Programs are usually sub-units of organizations, agencies or departments. Some program are large and have several self-contained COMPONENTS. FOR THE PURPOSE OF YOUR ASSIGNMENT, YOU NEED TO BE EVALUATING EITHER A SIMPLE PROGRAM OR A PROGRAM COMPONENT They should address a specific problem or problems, have clearly defined 1. purpose, 2. objectives, 3. services, 4. activities-which are components of the services and 5. sources of data that can be used for analysis. Data sources should be usable in order to measure whether or not a program is meeting its objectives. Objectives should be 1. clear, 2. specific and 3. measurable. Data should be either available or obtainable in order to measure objectives!!!
Every program should have 1. A purpose= a way to solve a need 2. A way to accomplish it 3. A way of knowing if this was done 1.A purpose = mission -> goals -> objectives -> program -> results (outcomes) (notice how these flow logically from one to the next) 2. A means of LOGICALLY achieving the above = resources (inputs) -> plan/strategies -> activities-> finishes (outputs) -> results (notice the logical structure here as well) 3. In the form of measuring meaningful outcomes and impact
You will find many programs that have major flaws in their logical structure. Three common one are: mission -> goals -> objectives -> program -> results (outcomes) resources (inputs) -> plan/strategies -> activities-> finishes (outputs) -> results measurement of meaningful outcomes and impact No specific measurable objectives Activities don’t Match purpose No real measurement of outcomes
You will probably run into one or more of those three in your program eval. When you do, you must really address it!!!!
DIFFERENCE BETWEEN SERVICES AND PROGRAMS Services What helping professionals offer (along with goods) in order to alleviate or eliminate a problem of an individual client or client system. Services may consist of various types of intervention including treatment, referral, case management, family assessment, and so on. Services are offered by one or more professionals, and are only one component of a program. Programs Subunits of organizations constructed in a unique manner to address a social problem. Programs are more or less self-contained and have their own goals(THAT MAY BE A PART OF THE ORGAINZIATION BUT THAT THE PROGRAM IS DESIGNED TO ADDRESS), objectives, policies, rules, procedures, services, strategies, activities, interventions, staff, budget, space, resources, and so on Programs may be time-limited or a relatively permanent part of an organization. Programs should be consistent with the organization’s mission, objectives and vision statements.
******The most important program components for a successful evaluation are: 1. specific problem or problems to be addressed 2. specific and measurable objectives specific and observable activities, interventions and/or services. 3. available, obtainable or creatable sources of data and data.
Example of a major problem in program evaluation: the program’s activities are not designed to meet the objectives! One of the major issues in evaluating a program is whether or not the activities, services and interventions are congruent with or help achieve the purpose and objectives of the program. This is sometimes NOT the case. For example, there is a day program for people with ‘chronic and severe psychiatric diagnoses. The purpose is to increase independent living. Three of the major objectives are to increase self-sufficiency in the daily living skills of 1. shopping, 2. money management and 3. home care. However, the activities that are observable and documentable are these: 1. Daily exercise walks. 2. jig saw puzzle working. 3. a group in which problems of daily living are discussed. It should be clear that the only activity that is directed toward the objectives is the group!!!!! UNLESS!!!!
1.THERE IS EVIDENCE IN THE LITERATURE ON SUCH PROGRAMS THAT EXERCISE WALKS AND JIG-SAW PUZZLE WORKING THAT SHOW THAT THEY HELP TO CREATE INDEPENDENT LIVING SKILLS!!! 2.Before you can do a program evaluation, you must FIRST RESEARCH WHAT HAS BEEN DOCUMENTED TO WORK AS A SOLUTION FOR THE SPECIFIC PROBLEM (ONCE YOU KNOW WHAT THE PROBLEM THAT THE PROGRAM IS ADDRESSING)! 3.This means you must review the literature on the problem in order to determine what has proven to be effective!!!! 4.You must do a literature review for the problem 5. A literature review is a systematic search of journals and books that may contain information on a specific problem and “evidence-based” solutions to it.