1. URBAN INSTITUTE Justice Policy Center National Youth Court Center Evaluation Workshop Janeen Buck Jeffrey Butts October 23, 2002 National Youth Court Seminar on Funding and Evaluating Teen Courts - Indianapolis, IN

2. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 5.0 Data Collection and Measurement Data –Qualitative vs. Quantitative –Sources Measurement –constructing questions –surveys and questionnaires Data collection –logistics and strategies –legal considerations

3. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 5.1 Data Sources and Methods Official Records (police, court, etc.) Face-to-face interviews Written surveys BenefitsProblems Often in database form Measures of official recidivism and program compliance Allows for large samples Database structures can be complex Restricted access Recidivism measures are biased Rich, personal data Measures of subjective perceptions and attitudes Useful for theoretically oriented evaluations Very expensive Requires prof. experience Increases respondent resistance and attrition Incentives ($$) needed Can be highly objective Measures of subjective perceptions and attitudes Good response rates Requires good design Requires prof. experience Limited range of variables

4. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 5.2 Measurement Issues Relevance and precision Simplicity Balance “Road-test” measures

5. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 Question Wording is an Art Question #1: I can always talk to my teachers about problems with my school work. Problem: Terms such as "always" and "never" create response bias. If respondents can talk to teachers 90% of the time, which answer should they choose? Strongly Don’t Strongly Disagree Disagree Know Agree Agree SD D DK A SA

6. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 Question Wording is an Art Question #2: Community work service and restitution payments are effective with youth. Problem: Which part are respondents answering, community service or restitution? They may agree with one and not the other, but the evaluator will never know. The question becomes useless from a research point of view. Strongly Don’t Strongly Disagree Disagree Know Agree Agree SD D DK A SA

7. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 Question Wording is an Art Question #3: The courtroom is hectic during teen court. Problem: Question employs a negative image. Respondents may think their courtroom is busy, but not “hectic.” Researchers won’t know what a “disagree” response actually means. Strongly Don’t Strongly Disagree Disagree Know Agree Agree SD D DK A SA

8. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 5.3 Data Collection, 5.4 Consent Developing a plan MOUs Informed consent –What? –Why? –When?

9. URBAN INSTITUTE Justice Policy Center National Youth Court Center: Evaluation Workshop October 2002 6.0 Working with an Evaluator Objectives and expectations Selection process –Request for Proposal (RFP) “Goodness of fit” Products and services