1. Randi Korn and Stephanie Downey Randi Korn & Associates, Inc. www.randikorn.com Evaluation Workshop October 10, 2013 SENCER-ISE Civic Engagement Partnership Awards Cohort 2

2. Workshop Goal Clarify audience outcomes, develop audience indicators, and choose data collection methods.

3. Cycle of Learning ACT PLAN OUTCOMES EVALUATE REFLECT What impact do we want to achieve? Do our actions align with the impact we want to achieve? What have we learned? How can we improve? In what ways have we achieved impact?

4. Outcomes: Common Pitfalls 1.Describing what you do instead of the result of what you do. 2.Including two separate ideas in one outcome statement. 3.Repeating an idea under different outcome types. 4.Identifying too broad and general of an audience. 5.Using vague, non-specific language to define terms or ideas.

5. Prioritizing Outcomes

6. 1. Which outcomes are most important to achieve (for you, for funders, for other stakeholders)? 2. Which outcomes are most realistic to achieve, given the scope and scale of your project?

7. Contextualizing Indicators

8. Outcomes/Indicators in Informal Learning Settings Outcomes are specific results of a program on distinct audiences; they actualize impact, make it explicit / concrete. Indicators describe the evidence and enable outcomes to become observable and measurable; they allow for consistency in evaluation.

9. Outcomes/Indicators in Informal Learning Settings NSF Impact Framework How Visitors Think The Burke’s Intentions

10. Example: Indicators Outcome: Participants will understand relationships that occur within a watershed.  Participants describe a watershed as the area of land that “collects” water that goes to the river.  Participants will know that everything that lives in a watershed, including people, is inter-related and dependent on one another.  Participants will identify negative impacts of construction and urban development, such as poor water quality, on the Rio Grande of Manati.

11. What must audiences do or say to demonstrate they have achieved an outcome?

12. Developing Indicators

13. Is the indicator concrete and specific? Is the indicator realistic to achieve given what you envision for the project?

14. Choosing Methods

15. Data Collection Methods Typical options  Standardized questionnaires (numerical information)  In-depth, open-ended interviews (descriptive information)  Observations (numerical and descriptive information)  Focus groups (descriptive information)

16. Standardized Questionnaires Produce quantitative data  Demographic characteristics  Attitudes about the program/concept  Expectations  Quality of experience

17. In-depth Interviews Produce qualitative data  Descriptive, detailed information about program experiences  Reveal nuanced understandings of ideas and concepts  Explore the meaning audiences construct from an experience

18. Observations Provide quantitative or qualitative data  Document an objective account of behavior  Reveal holding power of certain experiences  Explore behaviors (e.g., asking questions, scaffolding)  Explore staff-user interactions (e.g., who initiates the interaction)  Use of specific components (e.g., handouts, program elements)

19. Focus Groups Produce qualitative data  Provide insight about why people think the way they do  Provide opportunities for people to discuss ideas with others  Generates and brainstorms ideas around specific ideas/concepts

20. Cycle of Learning ACT PLAN OUTCOMES EVALUATE REFLECT What impact do we want to achieve? Do our actions align with the impact we want to achieve? What have we learned? How can we improve? In what ways have we achieved impact?