1. September 2007 Survey Development Rita O'Sullivan Evaluation, Assessment, & Policy Connections (EvAP) School of Education, University of North Carolina-Chapel Hill ritao@unc.edu or 919/843-7878

2. September 2007 Overview Evaluation Theory & Practice Elements of an Evaluation Plan: Logic Models Evaluation Instruments Q & A

3. September 2007 Surveys for a Purpose What information is needed? Who needs the information? How will it be used? Who has the information you need? What’s the best way to get it? What resources are available?

4. September 2007 Evaluation Planning - Logic Models Goals and Objectives Activities Performance Measures/ Outcomes Monitoring Evidence Outcome Evidence What do you want to do? What purposes does the program serve? What does it seek to accomplish? Activities to implement to achieve the stated goals & objectives Expected outcomes or results for each activity. Answers the question “Why are we implementing this particular activity? What do we expect the outcome to be?” Language in this column often reflects change: increase x or decrease y. Evidence of Activities & Quality Instruments and forms used to collect data which will demonstrate that the activities have occurred Evidence of Results Evidence they collected to demonstrate that the specified outcomes have been achieved

5. September 2007 Guide to Instrument Selection Data Collection StrategyKey Advantages Knowledge tests:Assess content knowledge  Limited responseCan cover large amount of content domain  Open-endedCan assess higher order cognitive skills  Performance AssessmentCan assess actual behaviors Opinion Surveys:Assess opinions  WrittenWritten record of responses On-SiteUsually good response rates MailCan contact people from different geographic areas EmailEase of survey distribution Web-BasedEase of survey distribution and data summary  TelephoneCan contact people from different geographic areas  GroupCost effective; Can assess degree of consensus  Face-to-FaceCan probe for more detail; Respondents can ask questions Attitude & Interest Surveys:Assess degree of attitude or interest; Easy to score

6. September 2007 Logic Model –All Stars GOALS & OBJECTIVES ACTIVITIES Performance Measures/ Outcomes MONITORING EVIDENCE OUTCOME EVIDENCE Decrease high school students drug use Introduce All Stars, Sr. curriculum in HS health education classes Students will report increased awareness of problematic substance abuse Student survey Student attendance records 75% of students will report reduced drug use as measured by pre-post survey 90% of students will report increased awareness of substance abuse as measured by pre-post survey Improve drug education in HS health ed classes Introduce All Stars, Sr. curriculum in HS health education classes Increase teacher capacity to conduct effective drug education curriculum Video analysis of classroom lessons Lesson plan logs 90% of participating teachers will use All Stars Curriculum

7. September 2007 Validity and Reliability o VALIDITY: Is the degree to which the assessment is appropriate to its purpose. o RELIABILITY: Will this assessment get the same response every time. o TRIANGULATION: Verifying findings through multiple data sources.

8. September 2007 Sampling Need for Explanation/ Rich Description Issue Complexity Difficulty around Obtaining Information QUALITATIVE DATA COLLECTION QUANTITATIVE DATA COLLECTION LOW HIGH Lack of Information about the Topic

9. September 2007 Types of Sampling Simple Random Stratified Cluster Systematic Purposeful Convenience Snowball

10. September 2007 Response Rates Factors that influence response rates Saliency Reminders Ease of response Increasing sample size will not increase response rates

11. September 2007 Ensuring Data Quality Sample representativeness? Consistent across individuals? Appropriate to purpose? Collected in a culturally appropriate manner?

12. September 2007 Draft Program Student Survey 6. How strongly do you agree or disagree with each of the following? Strongly disagree Disagree Neither agree or disagree Agree Strongly agree a. This program was excellent. OOOOO b. This program helped me prepare for a career and life-long learning. OOOOO c. This program didn’t decreased my interest. OOOOO d. Because of this program I have improved my skills. OOOOO e. I would recommend this program to others. OOOOO

13. September 2007 Sample Focus Group Protocol SCALE wants to know how the Student to Student program is working and what impact it is having on your campuses and with your community partners. They have asked us to conduct this focus group to begin exploring these areas. After we talk, we will summarize what you’ve said to capture the essential points of our discussion; we will make sure that individuals cannot be identified from the summary. Before we share the summary with SCALE, we will share it with you to make sure that what we’ve summarized fits with what you intended to say. If no one objects we’d like to record this focus group so that we won’t lose any detail of the discussion. Start-Up/Campus Recruitment What kinds of expectations are realistic and reasonable in terms of initial campus start up? What barriers to recruitment of community partner(s) did campuses face? What barriers to recruitment of tutors did campuses face? Tutor Training: What worked in terms of the initial tutor training SCALE did on your campuses? How can this initial tutor training be improved? What kinds of expectations are realistic and reasonable in terms of initial tutor training? What ongoing training do tutors need? What ongoing tutor training opportunities are being offered at campus sites? How, if at all, is the tutoring manual being used during ongoing tutor training?