1. Measuring Outcomes and Designing Research

2. 6/6/2019
What is an Outcome?
Characteristic of the target population or the social condition that a program or policy is expected to change
Does the use of body worn cameras by the police lead to an increase in individuals’ level of trust in the police?
Does increasing police patrol at crime hot spots lead to a reduction in crime?

3. Identifying Outcomes: Logic Models with Proximal and Distal Effects
6/6/2019
Identifying Outcomes: Logic Models with Proximal and Distal Effects
Express outcomes as part of a logic model connecting program activities to proximal (intermediate) outcomes that, in turn, are expected to lead to other, more distal, outcomes

4. Example of a Logic Model
6/6/2019
Example of a Logic Model
Training on effective strategies for interviewing sexual assault victims
Program Activity
More compassionate treatment of sexual assault victims
Proximal Outcome
Victims less likely to refuse to cooperate with police
Increase in arrests, prosecutions and convictions
Distal Outcomes

5. Measuring Outcomes Outcome: Reduce youth involvement in crime
6/6/2019
Measuring Outcomes
Outcome: Reduce youth involvement in crime
How to measure?
Whether youth engages in any delinquent behavior (prevalence)
Number of delinquent acts (frequency)
Severity of delinquent acts
Time to first new delinquent act

6. Measuring Outcomes Outcome: Increase police effectiveness
6/6/2019
Measuring Outcomes
Outcome: Increase police effectiveness
How to measure?
Crime rate(s)
Arrest rates
Clearance rates
Response time
Number of citizen complaints about police
Satisfaction with police
Trust in police

7. Things to Consider in Developing Outcome Measures
6/6/2019
Things to Consider in Developing Outcome Measures
Is program/policy change designed to affect individuals, organization, community, or society as a whole?
Are the measures reliable and valid?

8. 1st Issue: Who/What is Program/Policy Designed to Affect?

9. Effects on Individuals
6/6/2019
Effects on Individuals
Program designed to affect individuals’
Attitudes
Knowledge
Skills
Behavior

10. Effects on Individuals
6/6/2019
Effects on Individuals
Example: training program on effective strategies for interviewing sexual assault victims that is designed to
Change attitudes about victims of sexual assault
Provide information about post-incident behavior of sexual assault victims
Teach skills that officers can use in interviewing victims
Increase arrests of suspects in sexual assault cases

11. Effects on the Community
6/6/2019
Effects on the Community
Example: anti-graffiti program designed to reduce social disorganization and empower community residents
Example: community policing initiative designed to reduce overall crime by targeting public order (nuisance) crimes, such as prostitution, public drunkenness, loitering

12. 2nd Issue: Are Outcome Measures Reliable and Valid

13. 6/6/2019
Reliability
Reliability: extent to which measure produces the same results when used repeatedly to measure the same thing
A doctor’s scale is reliable to the extent that it reports the same weight for persons weighing exactly the same
Variation = measurement error

14. Are Crime Statistics a Reliable Measure of the Amount of Crime?
6/6/2019
Are Crime Statistics a Reliable Measure of the Amount of Crime?
Not all crimes are reported
Not all reported crimes are recorded
There are differences in the meaning of “crime,” the definitions of individual crimes, and the rules for classifying crimes

15. Assessing Measures for Reliability
6/6/2019
Assessing Measures for Reliability
Test-retest using same subjects
If measure is reliable, should produce the same results for each subject each time
Internal Consistency Reliability
Consistency of responses to similar items in a multi-item measure administered at same time
Inter-rater Reliability
Degree to which two individuals record data on the outcome in the same way

16. 6/6/2019
Validity
Validity: extent to which the indicator measures what it is intended to measure
Are we measuring what we think we are measuring?
Are arrests for drug offenses a valid indicator of level of community drug use?
Are arrest rates or crime rates or clearance rates valid indicators of the effectiveness of the police?

17. Assessing Measures for Validity
6/6/2019
Assessing Measures for Validity
Expect roughly same results if two measures of same outcome are applied at same time
Two different measures of trust in the police should produce similar results
Expect different results when measure is applied to individuals or situations that are different
Measure of trust in the police should produce different scores for individuals in high-crime and low-crime communities

18. Reliability Versus Validity
6/6/2019
Reliability Versus Validity

19. Types of Research Designs
6/6/2019
Types of Research Designs
Experimental
Randomized control trial—the “gold standard”
Quasi-Experimental
Non-Experimental

20. Experimental Design Ideal Design for Measuring Impact Key elements:
6/6/2019
Experimental Design
Ideal Design for Measuring Impact
Key elements:
Random Assignment to groups
Program group
Control group
Before and After Measurement

21. 6/6/2019
Experimental Design
R O1 X O2 R O1 O2 R indicates Random assignment O is the Observation or measure X is the Program or the intervention

22. Classic Experimental Design: Testing for Program/Policy Impact
6/6/2019
Classic Experimental Design: Testing for Program/Policy Impact

23. Example: Minneapolis Domestic Violence Experiment (1983)
6/6/2019
Example: Minneapolis Domestic Violence Experiment (1983)
Complaint regarding domestic violence led to one of three randomly assigned responses by the police
Send assailant away from home for 8 hours
Mediate/arbitrate dispute/counsel the parties
Arrest the assailant

24. Example: Minneapolis Domestic Violence Experiment
6/6/2019
Example: Minneapolis Domestic Violence Experiment
Outcome measure: rearrest (same suspect/same victim) within 6 months
Results
Arrest—10%
Mediate/Counsel—19%
Send assailant away—24%

25. Other Research Designs
6/6/2019
Other Research Designs
Other evaluation designs
Quasi-experimental
Non-experimental
Quasi-experimental designs are weaker than experimental and non-experimental designs are weaker than quasi- experimental
More vulnerable to incorrect interpretations of project impacts

26. Quasi-Experimental Design
6/6/2019
Quasi-Experimental Design
O X O2 Program Group
O O2 Control Group
Program Group and Control Group but
No random assignment
Matched pairs
Non-equivalent comparison groups

27. Quasi-Experimental Design
6/6/2019
Quasi-Experimental Design
Study of the effect of increased police patrol on violent crime rates in a city’s neighborhoods
Each neighborhood matched with another neighborhood on crime rate, poverty rate, unemployment rate and other relevant factors
One neighborhood received increased police patrol for six months, other did not

28. Two Examples of Non-Experimental Designs
6/6/2019
Two Examples of Non-Experimental Designs
Before and After Design (lacks control group):
O X O
Measure violent crime rate 6 months before and 6 months after an increase in police patrol
Interrupted Time-Series Analysis
O O O O O O X O O O O O O
Measure violent crime over time to determine if an increase in police patrol led to a decline in violent crime

29. 6/6/2019
Survey Research
Collecting information from individuals by asking them standardized questions or asking them to respond to or evaluate hypothetical situations
In person interviews
Questionnaires
Telephone or mail/online surveys

30. 6/6/2019
Survey Research
Question: Do men and women (or youth and adults) have different levels of trust in the police?
Survey a sample of individuals and compare responses of men and women (or of youth and adults)

31. Trust in the Police and Age
6/6/2019
Trust in the Police and Age

32. National Youth Survey (NYS)
6/6/2019
National Youth Survey (NYS)
Longitudinal survey of delinquent behavior and alcohol/drug use among American youth (11 to 17 years of age)
Self-report questionnaires

33. National Youth Survey: Drug Use Over Time
6/6/2019
National Youth Survey: Drug Use Over Time