Measuring Outcomes and Designing Research

6/6/2019 What is an Outcome? Characteristic of the target population or the social condition that a program or policy is expected to change Does the use of body worn cameras by the police lead to an increase in individuals’ level of trust in the police? Does increasing police patrol at crime hot spots lead to a reduction in crime?

Identifying Outcomes: Logic Models with Proximal and Distal Effects 6/6/2019 Identifying Outcomes: Logic Models with Proximal and Distal Effects Express outcomes as part of a logic model connecting program activities to proximal (intermediate) outcomes that, in turn, are expected to lead to other, more distal, outcomes

Example of a Logic Model 6/6/2019 Example of a Logic Model Training on effective strategies for interviewing sexual assault victims Program Activity More compassionate treatment of sexual assault victims Proximal Outcome Victims less likely to refuse to cooperate with police Increase in arrests, prosecutions and convictions Distal Outcomes

Measuring Outcomes Outcome: Reduce youth involvement in crime 6/6/2019 Measuring Outcomes Outcome: Reduce youth involvement in crime How to measure? Whether youth engages in any delinquent behavior (prevalence) Number of delinquent acts (frequency) Severity of delinquent acts Time to first new delinquent act

Measuring Outcomes Outcome: Increase police effectiveness 6/6/2019 Measuring Outcomes Outcome: Increase police effectiveness How to measure? Crime rate(s) Arrest rates Clearance rates Response time Number of citizen complaints about police Satisfaction with police Trust in police

Things to Consider in Developing Outcome Measures 6/6/2019 Things to Consider in Developing Outcome Measures Is program/policy change designed to affect individuals, organization, community, or society as a whole? Are the measures reliable and valid?

1st Issue: Who/What is Program/Policy Designed to Affect?

Effects on Individuals 6/6/2019 Effects on Individuals Program designed to affect individuals’ Attitudes Knowledge Skills Behavior

Effects on Individuals 6/6/2019 Effects on Individuals Example: training program on effective strategies for interviewing sexual assault victims that is designed to Change attitudes about victims of sexual assault Provide information about post-incident behavior of sexual assault victims Teach skills that officers can use in interviewing victims Increase arrests of suspects in sexual assault cases

Effects on the Community 6/6/2019 Effects on the Community Example: anti-graffiti program designed to reduce social disorganization and empower community residents Example: community policing initiative designed to reduce overall crime by targeting public order (nuisance) crimes, such as prostitution, public drunkenness, loitering

2nd Issue: Are Outcome Measures Reliable and Valid

6/6/2019 Reliability Reliability: extent to which measure produces the same results when used repeatedly to measure the same thing A doctor’s scale is reliable to the extent that it reports the same weight for persons weighing exactly the same Variation = measurement error

Are Crime Statistics a Reliable Measure of the Amount of Crime? 6/6/2019 Are Crime Statistics a Reliable Measure of the Amount of Crime? Not all crimes are reported Not all reported crimes are recorded There are differences in the meaning of “crime,” the definitions of individual crimes, and the rules for classifying crimes

Assessing Measures for Reliability 6/6/2019 Assessing Measures for Reliability Test-retest using same subjects If measure is reliable, should produce the same results for each subject each time Internal Consistency Reliability Consistency of responses to similar items in a multi-item measure administered at same time Inter-rater Reliability Degree to which two individuals record data on the outcome in the same way

6/6/2019 Validity Validity: extent to which the indicator measures what it is intended to measure Are we measuring what we think we are measuring? Are arrests for drug offenses a valid indicator of level of community drug use? Are arrest rates or crime rates or clearance rates valid indicators of the effectiveness of the police?

Assessing Measures for Validity 6/6/2019 Assessing Measures for Validity Expect roughly same results if two measures of same outcome are applied at same time Two different measures of trust in the police should produce similar results Expect different results when measure is applied to individuals or situations that are different Measure of trust in the police should produce different scores for individuals in high-crime and low-crime communities

Reliability Versus Validity 6/6/2019 Reliability Versus Validity

Types of Research Designs 6/6/2019 Types of Research Designs Experimental Randomized control trial—the “gold standard” Quasi-Experimental Non-Experimental

Experimental Design Ideal Design for Measuring Impact Key elements: 6/6/2019 Experimental Design Ideal Design for Measuring Impact Key elements: Random Assignment to groups Program group Control group Before and After Measurement

6/6/2019 Experimental Design R O1 X O2 R O1 O2 R indicates Random assignment O is the Observation or measure X is the Program or the intervention

Classic Experimental Design: Testing for Program/Policy Impact 6/6/2019 Classic Experimental Design: Testing for Program/Policy Impact

Example: Minneapolis Domestic Violence Experiment (1983) 6/6/2019 Example: Minneapolis Domestic Violence Experiment (1983) Complaint regarding domestic violence led to one of three randomly assigned responses by the police Send assailant away from home for 8 hours Mediate/arbitrate dispute/counsel the parties Arrest the assailant

Example: Minneapolis Domestic Violence Experiment 6/6/2019 Example: Minneapolis Domestic Violence Experiment Outcome measure: rearrest (same suspect/same victim) within 6 months Results Arrest—10% Mediate/Counsel—19% Send assailant away—24%

Other Research Designs 6/6/2019 Other Research Designs Other evaluation designs Quasi-experimental Non-experimental Quasi-experimental designs are weaker than experimental and non-experimental designs are weaker than quasi- experimental More vulnerable to incorrect interpretations of project impacts

Quasi-Experimental Design 6/6/2019 Quasi-Experimental Design O1 X O2 Program Group O1 O2 Control Group Program Group and Control Group but No random assignment Matched pairs Non-equivalent comparison groups

Quasi-Experimental Design 6/6/2019 Quasi-Experimental Design Study of the effect of increased police patrol on violent crime rates in a city’s neighborhoods Each neighborhood matched with another neighborhood on crime rate, poverty rate, unemployment rate and other relevant factors One neighborhood received increased police patrol for six months, other did not

Two Examples of Non-Experimental Designs 6/6/2019 Two Examples of Non-Experimental Designs Before and After Design (lacks control group): O X O Measure violent crime rate 6 months before and 6 months after an increase in police patrol Interrupted Time-Series Analysis O O O O O O X O O O O O O Measure violent crime over time to determine if an increase in police patrol led to a decline in violent crime

6/6/2019 Survey Research Collecting information from individuals by asking them standardized questions or asking them to respond to or evaluate hypothetical situations In person interviews Questionnaires Telephone or mail/online surveys

6/6/2019 Survey Research Question: Do men and women (or youth and adults) have different levels of trust in the police? Survey a sample of individuals and compare responses of men and women (or of youth and adults)

Trust in the Police and Age 6/6/2019 Trust in the Police and Age

National Youth Survey (NYS) 6/6/2019 National Youth Survey (NYS) Longitudinal survey of delinquent behavior and alcohol/drug use among American youth (11 to 17 years of age) Self-report questionnaires

National Youth Survey: Drug Use Over Time 6/6/2019 National Youth Survey: Drug Use Over Time