Evaluation research Using research methods in combinations Policy analysis

Evaluation  Examining the effects of programs, policies and laws  Programs: providing services to clients  Policies: rules for handling clients, actions taken i.e., policy on deadly force for fleeing felons  Laws: what are the real impacts

Evaluation  People assumed that particular effects would take place  However, those effects may or may not happen  There may be unanticipated side effects to a change, both good and bad

Policy  Usually some issue is raised  Demands are made, and there is support and opposition  Goals should be formulated and ways established to attain those goals (objectives)  Policy output: the program, resources allocated

Policy  Then the impact is determined  Did the changes in policy outputs have an impact on the problem?  Hypothesis: If some policy action is taken, then we expect a particular result  To test the hypothesis, must do an evaluation

Evaluation components  Process evaluation and impact  Process  Needs assessment and monitoring part of the process evaluation  The first step should be needs assessment  Frequently this is not done

Needs assessment  Is the program/policy/intervention needed?  Is the problem serious? Will it “get better” on its own?  Who are the members of the target population? (those most in need of services, deterrents, etc.)  What are their characteristics?

Needs assessment  Programs will have little impact if they don’t affect the target population  Programs will have little effect if they do not take into account the characteristics of the target population

Needs assessment  i.e., a school based program will not help chronically truant delinquents  Adding 11 th and 12 th grade classes at a reform school will not be useful if very few delinquents at the school are functioning at even a 9 th grade level  Welfare to work requires child care

Needs assessment  There is a need to gather information to determine the characteristics of the target population  May not be readily apparent, as the target population might not be in the system  Assessment methods: records, key informants, gatekeepers, data collection

Needs  If the target population is not reached, even good programs, policies and laws may not have an impact  Weed and Seed example  Adolescent drug offender example

Needs  Establishment of goals and objectives  Exercise: what are the goals of a prison  Mechanisms (objectives, activities) for meeting those goals  Develop a list

In an evaluation  The research must ask  What does the program intend to accomplish?  How do existing staff determine whether they have attained their goals?  Have formal goals and objectives been identified and prioritized?

Evaluation  What performance measures are currently used?  Are they adequate, or must additional measures be developed as part of the evaluation?  It would be very useful if evaluators were included at the start of a program to establish measures, seldom happens

Monitoring (process)  What is actually happening in the intervention?  What happens on paper, theoretically, may be different from what actually happens  Who is actually served by the program?  Are they members of the target population?

Monitoring  If they are not, the program will probably have little impact  how are people selected for inclusion, or how do they decide to participate?  How do participants differ from members of the target population?  How do potential participants differ from nonparticipants?

Monitoring  How do staff feel about the program or policy?  Example: laws and prosecutors  Juvenile judges and juvenile justice programs  Is the program or policy actually being implemented?

Monitoring  Are all the components being addressed?  Example of drug courts  Program components and benchmarks, means of assessing whether program components are being met

Monitoring  Are there problems that were not anticipated?  i.e., lack of community support, resistance to the program or policy, reactions of criminal justice personnel, unexpected obstacles

Monitoring  Unanticipated side effects  Could be positive or negative  Positive: conditions improve for people not in the target population  Mental health centers, nutrition requirements for WIC  Negative: example of net widening  Program may do something else