Evaluation Designs and Methods Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D. Professor and Youth Development Specialist Oregon State University 4-H Professional Development Webinar February 14, 2013

Webinar Agenda Building on previous month’s topic of focusing and planning your evaluation we will: Explore the concept of rigor in evaluation, and its particular place in Extension and 4-H evaluation Learn the role that evaluation questions play in determining evaluation design and data collection methods Explore common evaluation designs Explore common and innovative evaluation methods Learn of resources available to support evaluation efforts in 4-H and Extension

Elements of Rigor Evaluation design Conceptualization of program constructs & outcomes Measurement strategies Timeframe of the evaluation study Program integrity Program participation and attrition Statistical analyses Braverman, M. T., & Arnold, M. E. (2008). An evaluator’s balancing act: Maintaining rigor while being responsive to multiple stakeholders. In M. T. Braverman, M. Engel, R. A. Rennekamp, & M. E. Arnold (Eds.) Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120,

Rigor and the 4-H Organization Who determines standards of rigor? How do decisions about evaluation methods get made? How, and to what extent, is the quality of a completed evaluation determined ?

(XO)(XO) O = “Observation” (data collection)X = “intervention” (program) Post Only Design Evaluation Question Example: What skills do campers report developing at science camp? TIME

Never Sometimes UsuallyAlways I can use scientific knowledge to form a question 1234 I can ask a question that can be answered by collecting data 1234 I can design a scientific procedure to answer a question 1234

Post Only Control Group Design E (XO) C (XO) Evaluation Question Example: Do youth who attend 4-H summer science camp have better science skills than youth who do not attend? O = “Observation” (data collection)X = “intervention” (program) E = Experimental group (program participants) C = Control group (non-participants ) TIME

What is the level of rigor? What cannot be said? Never Sometimes UsuallyAlways I can use scientific knowledge to form a question 1234 I can ask a question that can be answered by collecting data 1234 I can design a scientific procedure to answer a question 1234

One Group Pre-Test/Post-Test (O XO) Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning? TIME

Strongly DisagreeDisagreeAgree Strongly Agree I feel good about my scholastic abilityOOOO I feel accepted by my friendsOOOO I can figure out right from wrongOOOO I can do things that make a differenceOOOO What is the level of rigor? What cannot be said?

Retrospective Pre-Test ( O XO) Evaluation Question Example: Do youth have higher levels of positive youth development at the end of the program than they did at the beginning? TIME

For each of the following items, please indicate how you felt before participating in this program, and how you feel now after participating in this program. 1 = Strongly disagree 2 = Disagree 3 = Agree 4 Strongly Agree BeforeAfter I feel accepted by my friends OOOOOOOO I can figure out right from wrong OOOOOOOO I can do things that make a difference OOOOOOOO

Control Group Pre-Test/Post Test Evaluation Question Example: Do youth in program develop higher levels of PYD than youth who do not participate? E (O XO) C (O ---O) Strongly DisagreeDisagreeAgree Strongly Agree I feel good about my scholastic abilityOOOO I feel accepted by my friendsOOOO I can figure out right from wrongOOOO I can do things that make a differenceOOOO

TIME LOW - Level of PYD - HIGH E C

Time Series Design with Control Group OOOOXOOOOOOOXOOO LOW - Level of PYD - HIGH E C

Choosing an Evaluation Data Collection Method Some Common Methods Archival data (records and documents) Surveys (mailed, electronic, phone) Interviews (phone, face to face, group) Focus group interviews Observation Tests (scenarios or skill/knowledge tests)

Important Steps What is the key concept that must be measured in each evaluation question? Did youth participants in the YA4-H! Teens as Teachers program pilot increase their own consumption of fruits and vegetables? Who has knowledge of this potential change? Several sources may emerge: Youth, parents, leaders, teachers, friends, records, observations. What sources of data will be acceptable to stakeholders? What expertise and funding is available to make a particular method practical?

Existing Documents and Records Have you ever considered meeting notes, minutes, videos, registrations, test scores, forms, records, reports as possible sources of data? Considerably more cost effective than original data collection Data are not affected by the act of collecting it Programs collect lots of information that is never used, and too often we forget to look for existing data that can answer the question What existing information could answer the question of increased fruits and vegetable consumption?

Surveys (Mailed, Electronic, Phone) How could surveys help us assess increase in youth consumption of fruits and vegetables? Designing Surveys: A Guide to Decisions and Procedures. Ronald Czaja and Johnny Blair (2005) Internet, Mail, and Mixed-Mode Surveys: Tailored Design Method (3 rd Ed.) John Dillman, Jolene Smyth, & Leah Melani Christian (2008) How to Conduct Surveys: A Step by Step Guide Arlene Fink (2013) Survey Research Methods (4 th Ed.) Floyd Fowler (2009)

Interviews (Phone, Face to Face, Group) Designing and Conducting Your First Interview (text book) Bruce Friesen (2010) Qualitative Interviewing: The Art of Hearing Data (3 rd Ed.) Hebert and Irene Rubin (2011) Focus Groups; A Practical Guide for Applied Research (2 nd Ed.) Richard Krueger (1994) Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences (4 th Ed.) Irving Seidman (2012) Why might interviews be a good method for collecting data about youth consumption of fruits and vegetables?

Direct Observation Situations where you want direct information May be more reliable than asking people if they are using new practices When you are trying understand an ongoing behavior, process or unfolding situation or event Observing camp counselors before, during and after a training program When there is physical evidence, products or outcomes that can be readily seen Inspecting project records, newsletters, signs When written or other data collection procedures seem appropriate Programs to vulnerable or underserved audiences, when language or literacy is a problem Key Resource Collecting Evaluation Data: Direction Observation Ellen Taylor-Powell and Sara Steele- Booklet available on the State 4-H Website Could we use direct observations to measure youth consumption of fruits and vegetable?

Tests and Scenarios Useful when assessing learning that requires specific knowledge that must be turned into action for the program to be considered successful. Requires the ability to do a real pre –post test process Can be very creative! Think skits and role plays Camp counselor training Youth leadership programs Risk management training Can you think of how a scenario evaluation might be a useful method for assessing youth consumption of fruits and vegetables?

Summary Stay tuned on March 6 th for our next webinar, which will focus more in depth on creating high quality questionnaires! Your Evaluation Question Determines Your DesignDetermines Your Methods

That’s all for now! Join in next month for: Creating High Quality Questionnaires That’s all for now! Join in next month for: Creating High Quality Questionnaires Don’t forget to complete an evaluation of today’s webinar at: