1 MODEL ACADEMIC CURRICULUM MODULE 13 Assessing and Evaluating Responses

2 Module 13 Components Assessment and Evaluation Conducting Community Surveys

3 Introduction 1.Did the problem decline? 2.If so, did the response cause the decline? The purpose of assessing a problem-solving effort is to help you make better decisions by answering two specific questions:

4 Assessment and Evaluation

5 Reviewing the SARA Model S CANNING A NALYSIS R ESPONSE ASSESSMENT

6 The Role of Evaluation in Problem-Solving

7 Types of Evaluations Process Evaluations Impact Evaluations

8 Interpretation of Process and Impact Evaluations

9 Conducting Process Evaluations Observe response implementation Interview relevant parties Conduct focus groups Conduct surveys

10 Conducting Process Evaluations Record decisions and progress in meeting minutes Develop a timeline of important achievements Document everything!!

11 Process Evaluation Timeline

12 Problem Solving Project - Process Evaluation California Highway Patrol Problem - Traffic Collisions  Environmental surveys - Personal inspection of 20 miles of roadway.  Key recommendation - 24-hour headlights-on policy.

13 Conducting Impact Evaluations Measures –Quantitative Measures –Qualitative Measures –Measurement Validity –Selecting Valid Measures

14 Criteria for Claiming Causality –There Is a Plausible Explanation of How the Response Reduces the Problem –The Response and the Level of the Problem Are Related –The Response Occurs Before the Problem Declines –There Are No Plausible Alternative Explanations

15

16 Types of Evaluation Designs

17 Pre-Post Designs

18 Pre-Post Design with a Control Group

19 Interrupted Time Series Designs

20 Multiple Time Series Design

21 Practical Limitations to Interrupted Time Series Designs Measurement is expensive or difficult. Data are unavailable for many periods before the response. Decision-makers cannot wait for sufficient time to elapse after the response. Data recording practices have changed, making inter- period comparisons invalid. Problem events are rare for short time intervals, forcing you to use fewer, longer intervals.

22 Combining and Selecting Designs

23 Spatial Displacement of Crime or Disorder, and Spatial Diffusion of Crime Prevention Benefits

24 Problem-Solving Evaluation Checklist I.Early Considerations II.Process Evaluation III.Impact Evaluation IV.Evaluation Conclusions V.Overall Impact Evaluation Conclusions

25 I. Early Considerations A.What will the evaluation help you decide? B.Do you know the problem? C.Do you know how the response works?

26 II. Process Evaluation A.Did you implement the response? B.Did you implement enough of the response?

27 III. Impact Evaluation A.Do you need a control group? B.How often can you measure the problem? C.What type of evaluation design should you use? D.What type of control group do you need?

28 IV. Evaluation Conclusions A.What are your findings from the process evaluation? B.What are your findings from the impact evaluation?

29 V. Overall Impact Evaluation Conclusions 1.Did the problem decline after the response? 2.If the problem did decline, did it do so at a faster rate after the response than before the response? 3.If the problem did decline, can you rule out all other plausible explanations for the decline, other than the response? Use your list of differences between the response and control groups to help answer this question.

30 Conducting Community Surveys

31 General Use of Surveys CJ Researchers –to better understand crime and public fear of crime Social Scientists and Political Pollsters –to learn about social relations and predict future events Government Agencies –to predict economic trends and how people react to policy Police –to measure public opinion and operational effectiveness

32 Specific Uses for Community-based Surveys Gather information about public attitudes regarding police or crime Detect and analyze problems in neighborhoods Evaluate problem solving efforts and other interventions by taking baseline measures Control crime and fear of crime

33 Basic Sampling Designs Simple random sampling – ensures that everyone in the population has a chance to be included in the sampling bias Nonrandom sampling – sometimes is the best option available given the time and other limits How many will be sampled? - Sample size, sampling error Survey Process

34 –Mail surveys –Telephone surveys –In-person interviews –Internet-based surveys –Officer/Deputy or volunteer delivery –Surveys are filled out at a designated location Methods for Contacting Respondents

35 –General considerations –How to ask questions –Open ended vs. close ended questions –Designing the questionnaire Asking Questions

36 –Characteristics of the sample –Does the sample represent the population? –Making inferences about the population –Estimating relationships –Significance testing Analysis of the Data

37

38

39