1. PRESENTATION ADAPTED from one developed by Emily Rothman, ScD Boston University Designing pre- and post-tests

2. Agenda Collect sustainability plans Scheduling for rest of year Review of learning module Lesson: Outcome Evaluation

3. Schedule through June Webinars May session:  Wednesday 5/8 or Friday 5/10 from 10-1 possible? May 30 th 9am-Noon June 13 th 10am-1pm  Last session  Leadership will be invited  Light lunch

4. Learning Module Review What is the difference between process and outcome evaluation? How can process evaluation be useful? Are you already collecting process evaluation data? How? What types of information? Is there any process evaluation data that you aren’t collecting but would like to?

5. What are measures?

6. Lingo Concept / wordExplanation SurveyThis is the whole package—typically comprises multiple measures MeasureBunch of questions (or “items”) that together assess a construct; may comprise scales Questionnairesame thing as measure ScaleSub-section of a measure or instrument InventorySame thing as measure; Measure of typical performance TestSame thing as measure; Measure of best performance InstrumentSame thing as measure ItemSingle question on a scale

7. What does a measure do? Operationalizes a concept Gets people to answer in the best possible way “Best” means most reliable and valid Gives you something to compare with other research studies Ensures your response options can be analyzed

8. Operationalizing a concept Example: Fear of falling cannot be directly measured. It is a concept.

9. Operationalizing a concept Are we truly trying to figure out…:  How worried might fall?  How likely might fall?  How often think about falling?  How often hold stair railings?  How confident can perform without falling? All slightly different takes on the same thing

10. Operationalize… “Bullying” Being physically assaulted by a peer Being called names, or harassed repeatedly by one or more peers Being intentionally excluded from activities by peers Being harassed via email, text, or social media repeatedly by one or more peers

11. Why use a “measure” instead of a question? The set of questions do a better job that one single item would alone Why?  People try to outsmart surveys  People are inconsistent in answers  More is better

12. Example: self-esteem Do you have high self-esteem?

13. Example: self-esteem 1. On the whole, I am satisfied with myself 2. At times I think I am no good at all. 3. I feel that I have a number of good qualities. 4. I am able to do things as well as most other people. 5. I feel I do not have much to be proud of. 6. I certainly feel useless at times. 7. I feel that I'm a person of worth, at least on an equal plane with others. 8. I wish I could have more respect for myself 9. All in all, I am inclined to feel that I am a failure 10. I take a positive attitude toward myself.

14. Psychometrics of scales/measures Social scientists develop scales They test and re-test them with groups of people They determine how reliable and valid the measures are

15. Wording changes responses Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year?  65% said companies could avoid layoffs  22% said couldn’t avoid layoffs  15% had no opinion Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year, or do you think layoffs are unavoidable?  35% said companies could avoid layoffs  41% said couldn’t avoid layoffs  24% had no opinion Payne, 1951

16. What is the connection to logic model?

18. Right now… Look at your logic model Identify an outcome How will you measure it?  Using which existing measure? … OR  Write your own “measure”?  Write your own single-item question?

19. Where to get measures

20. Where do you find measures? Lit search for measures Lit search for articles on same topic, see what measures they used After you know the name of a measure, you can sometimes Google it Usually you have to find it printed in a peer review journal It is hard to find measures!

21. Don’t make up measures unless… Golden rule: Do not make up survey questions out of your own head, ever, unless you really, really, really have to Acceptable reasons for writing own question: Searched literature, none exist Searched literature, all measures are inappropriate for some reason You are never going to publish your results or compare with another existing source

22. Types of Instruments Likert-type scale Yes/no questions Free response or qualitative sections

23. Creating an entire survey

24. What goes into a survey? Standard things:  Version (pre- or post-test)  Today’s date  Subject ID number? Demographics  Race/ethnicity?  Sex?  Age? Measures  Measure short-term, intermediate and long-term outcomes? Nothing extra!  Don’t ask any questions that you don’t have to—long surveys are bad.

25. What makes unreliable and invalid questions? Respondent (R)… 1. Doesn’t know the answer 2. Cannot recall an accurate answer 3. Doesn’t understand the question 4. Doesn’t want to report an accurate answer

26. Question design pitfalls Putting two questions in the same question: Have you ever broken your arm or been too sick to go to class?

27. Question design pitfalls? Using technical jargon without defining it: How frequently do you advocate for prophylactic use during sexual encounters?

28. Question design pitfalls Leading respondents: Would you agree that this program is awful?

29. Question design Be wary of socially desirable or undesirable questions: “Do you like pancakes?” vs. “Have you ever been diagnosed with a STI?” “How much money did you earn last year?”

30. Question design Keep it simple Place the burden on yourself to make things clear, not the respondents to figure things out  “What was your pattern of enrollment?” vs.  “Were you a part-time or full-time student?” Keep questions short

31. Question design No double negatives! If you have not had sex in the past year, do not answer the next question

32. Formatting design Neat, clean appearance Easy to read Large enough font No fancy serifs or font style No grammar or spelling mistakes

34. Work Session! Begin designing your survey…

35. Homework Tighten up your survey Administer with at least 10 people Complete the webinar and accompanying homework

36. Wrap-up +/-/Change Be in touch for technical assistance!