2. How To Evaluate Programs/Interventions? Step 1 – Identify independent and dependent variables/concepts Independent – Intervention Dependent – Stated objective of program Step 2 – Define these variables/concepts Step 3 – Develop indicators (i.e., questions) to measure these variables/concepts

3. Defining Variables/Concepts Challenges with definitions  Concepts may be multi-dimensional  Concepts can be value laden  Concepts can have both a subjective and objective definition—you must choose one  Only concepts that VARY (are also variables) can be used

4. Examples of Multi-Dimensional Concepts Social class ◦ Social class has an economic, occupational and education component Prejudice ◦ Although we often speak of prejudice as one term, there are many different types of prejudice (i.e., racial prejudice, social class prejudice, religious prejudice, etc.). These types comprise different components of prejudice.

5. Examples of Value Laden Concepts Prejudice – not acceptable to have these types of attitudes ◦ Are you prejudiced? Child abuse – not socially acceptable to engage in these types of behaviors or admit having been a recipient ◦ Have you abused your child today? ◦ Have you ever been abused? Sexual assault – negative stigma is often associated with having been raped ◦ Have you ever raped anyone? ◦ Have you ever been raped?

6. Measurement Error Results from Value Laden Concepts Social desirability ◦ There are socially desirable answers to questions  Example – If you asked the question “Have you ever abused your child?” the socially desirable answer would obviously be no Acquiescence bias ◦ Positive vs. negative answers  Example – If people are evaluating a program and they feel connected or grateful to those in charge, they will positively evaluate every item Leading questions ◦ Questions that predispose respondents to answer a certain way  Example - Rape is a heinous crime. Do you agree?

7. Subjective vs. Objective Definitions of Concepts Social class – objective (as defined by others) ◦ Family income, level of education, employment Social class – subjective (as perceived by the respondents) ◦ Ask – What is your social class? ◦ Ask – Would you say you are poor?

8. When a Concept is a Variable, It can Vary across time Vary across place Vary across individuals or groups

9. Examples of Concepts that Vary Across time ◦ Crime rates vary across age ◦ Annual income varies across age/life span Across place ◦ Crime rates vary across rural vs. urban areas ◦ Median incomes vary across rural vs. urban areas Across individuals/groups ◦ Crime rates vary across gender ◦ Income varies across gender

10. Examples of Concepts that Do NOT Vary Across time ◦ Sex – if you are born a male, you will die a male ◦ Eye color – generally you have same eye color all your life Across place ◦ Impact of frost on plants – generally if it freezes, then plants die no matter where you plant them ◦ Whether or not babies are born with teeth does not vary across place ◦ Likelihood that children will be born speaking Chinese Across Individuals/Groups ◦ Impact of restful sleep – generally you are healthier if you sleep well - no matter who you are ◦ Natural hair color doesn’t vary across gender ◦ IQ scores don’t vary across gender

11. You should (1) take a concept, (2) define it as a variable, and (3) develop an indicator to measure it ConceptVariableIndicator IncomeAnnual earningsWhat was your total income? RaceSelf identification of RaceWhat race are you? Poverty CrimeNumber of Crimes Committed up to age 55 How many felonies have you committed in your lifetime? How many misdemeanors have you committed in your life time?

12. Variables can be measured at different levels—These include: LEVELDefinitionVariableExample Nominal Measured in categories. There is no rank order to the category GenderMeasured as male or female Ordinal There is a rank order, but the distance between ranks is not known. Placement of vegetables grown at county fair. Squash may be judged to be a grand prize winner, first place, second place, etc. Interval There is a rank order, and distance between levels is known. Income LevelAsk respondents to indicate their income category: 0- 19,999; 20,000-39,999 etc. Ratio There is a rank order, and the distance between levels is known. There is an absolute zero. AgeMeasured as number of years old. Ranges from 0 to about 107.

13. You should always measure variables at the most precise level possible LEVELDefinitionVariableExample Nominal No Rank Order Categorical GenderWhat is your gender (1) Male (2) Female Ordinal Rank Order Distance not Known Satisfaction with Services I am satisfied with your services. (1) SD (2) D (3) NAND (4) A (5) SA Interval Rank Order Distance has Meaning Education LevelHow many years of formal education do you have? (1)0 – 4 (2)5 – 9 (3)10 – 14 Ratio Rank Order Distance has No Meaning Absolute Zero Family SizeHow many children do you have living in your household?

14. The same variable can be measured at different levels—For example… LEVELDefinitionVariableExample Nominal No Rank Order Categorical AGEAre you a “baby boomer”? 1.Yes 2.No Ordinal Categories are rank ordered, but the distance between levels is not equal and not known. AgeHow old are you? 1.Young Adult 2.Middle aged 3.Elder Interval Rank Order Distance has Meaning AGEHow old are you? 1.20-29 2.30- 39 3.40- 49 etc. Ratio Rank Order Distance has No Meaning Absolute Zero AGEHow old were you at your last birthday? _____ years old.

15. Always measure variables at the highest level possible. For example, whenever possible measure age at the ratio level LEVELDefinitionVariableExample Nominal No Rank Order Categorical AGEAre you a “baby boomer”? 1.Yes 2.No Ordinal Categories are rank ordered, but the distance between levels is not equal and not known. AgeHow old are you? 1.Young Adult 2.Middle aged 3.Elder Interval Rank Order Distance has Meaning AGEHow old are you? 1.20-29 2.30- 39 3.40- 49 etc. Ratio Rank Order Distance has No Meaning Absolute Zero AGEHow old were you at your last birthday? _____ years old.

16. Now you try ConceptVariableIndicator Crop Yield# of bales of hay per acre Educational level# of high school graduates in group OR the # of years of education completed for participants. Satisfaction with 4-H Workshop Level of satisfaction with their experience. Leadership skillsSubjective evaluation of own leadership skills *Leadership skills is a multi-dimensional concept, and implies a combination of various skills. It would therefore be wise to measure it using multiple indicators.

17. Age – more detailed example Categorical/Nominal ◦ Are you a baby boomer? (1) yes (2) no Ordinal ◦ Which of the following would describe you?  Child  Teenagers  Young adult  Adult Interval ◦ How old are you?  10-14  15-19  20-24 Ratio ◦ How old were you at your last birthday? ___ years Always measure variable at highest level possible

18. Income – more detailed example Categorical/Nominal ◦ Are you a poor? (1) yes (2) no Ordinal ◦ Rank you income relative to other college students  Much below averageAverage  Below averageSomewhat above average  Somewhat below averageAbove average Much above average Interval ◦ What was your annual income for 2008?  Less than 10,00020,000 to 29,99940,000 to 49,999  10,000 to 19,999930,000 to 39,99950,000+ Ratio ◦ How much money did you make last year? _____ dollars ______ cents Always measure variable at highest level possible

19. Now you try ConceptVariableIndicator Crop Yield# of bales of hay per acre Educational level# of high school graduates in group OR the # of years of education completed for participants. Satisfaction with 4-H Workshop Level of satisfaction with their experience. Leadership skillsSubjective evaluation of own leadership skills *Leadership skills is a multi-dimensional concept, and implies a combination of various skills. It would therefore be wise to measure it using multiple indicators.

20. Possible Answers ConceptVariableIndicator Crop Yield # of bales of hay per acreAsk farmers – How many acres of hay per acre did you harvest this year? Educational Level # of high school graduates in group Ask high school administrators – How many of the participants graduated from high school? OR the # of years of education completed for participants. Ask participants – What is the highest level of education you have completed? Satisfaction with 4-H Workshop Level of satisfaction with their experience. Ask respondents – Indicate your degree of satisfaction with workshop Leadership Skills Subjective evaluation of own leadership skills Ask respondents – Please indicate the extent to which you agree with the following statements. 1.People listen when I talk 2.Others respect my suggestions 3. People think I have good ideas

21. Questions or comments? Please contact: Carol Albrecht Assessment Specialist USU Extension 979-777-2421 carol.albrecht@usu.edu