1. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Evaluating STEM Professional Development in Informal Education Settings Sheila A. Arens & Jesse Rainey

2. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Train-the-trainer approach: individuals from informal science education settings provided training on climate change and how to deliver information about climate change to the public. NASA Climate Day / Earth Ambassadors Trained trainers become Earth Ambassadors, expected to design and implement Climate Day events at their informal “home sites. ” Events typically were part of other events being held at the Earth Ambassadors’ institutions (lots of variation in delivery / implementation) NASA-sponsored

3. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation PD for informal educators aligned well with K-12 best practices in PD: ongoing and intensive connected to subject matter involved subject matter experts supported educators to develop strategies for eliciting prior knowledge encouraged use formative assessment information to guide instruction involved active learning Professional Development for Earth Ambassadors

4. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Supplemental Materials

5. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Recruited Sites

6. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Evaluation Questions & Data Collection Overarching Research Question Ancillary Questions Data Collection Method(s), Source(s) of Data & Instrument Development Needed What were the successes and challenges of the Earth Ambassadors training?  Are there changes in Earth Ambassador’s attitudes about climates following training?  What are Earth Ambassador’s experiences during and following training?  Surveys  Interviews (stratified sample of Earth Ambassadors regarding experiences during and following training)  Artifacts from Earth Ambassador trainings What were the successes and challenges of the Climate Day Events?  What are the perceptions of participants in the Climate Day Events?  What successes and challenges did Earth Ambassadors experience as they implemented their Climate Day Events?  Were Climate Day Events implemented as planned and what are the perceptions of participants regarding the Event? (sample)  What additional training or resources do Earth Ambassadors need to improve the successes or efficacy of their Events?  Participant surveys  Earth Ambassador survey (key Program staff)  Observations (sample two Climate Day Events)  Artifacts from Climate Day events

7. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Overarching Findings Earth Ambassadors perceived themselves to be more knowledgeable about climate change, ways to be “greener,” and strategies for using the materials. Earth Ambassadors considered themselves as more confident in presenting information to the public. Event participants noted an increased understanding of climate and climate change, increased confidence to talk with others about climate, and an interest in attending more climate-related events.

8. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Examining EA “Reach” Using Geocoding

9. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Focus on a “case” Client expressed interest in learning more from some of the sites. In collaboration with them, we determined to… conduct additional data collection about implementation at a sample of sites, and more closely example one of the sites where the “reach” of the program had the most potential. Kopernik Observatory & Science Center, Vestal, NY: This example is a realization of the Earth Ambassadors theory of change that moves from training to information to dissemination and to planned community action—and has led to positive steps toward addressing issues of climate change. Collect additional extant data (on the location and the politics / policies in the area) Collect additional data o Surveys, interviews o Observation

10. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation

11. September 2013: Conference on Extreme Weather, Climate Change, Mitigation and Adaptation: An “action-oriented” conference aimed at municipal officials in the Southern Tier of New York State. June 2014: “Public Night” Event: Focused on extreme weather and adaptation, and featured a speaker from a local university and winners of a student essay contest about climate change. September 2014: Director and Fire Coordinator for a county public safety facility spoke about the community’s experiences with the floods of 2006 and 2011 and lessons learned in preparing for the next flood. October 2014: Preparing New York’s Communities for Climate Change: Speaker from the New York State Department of Environmental Conservation will discuss recommendations from the New York State Office of Climate Change, including the key role local governments play in responding to climate change, and the adoption of the Climate Smart Communities pledge. Additional data collected at this site provided information on how the program impacts perceptions and planned actions among government leaders and outcomes within their respective communities. Site Activities & Additional Data Collection

12. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Observation Protocol for Climate Day Event Approximate number of attendees: Activity: Time: Total Minutes: List of materials and resources used (indicate whether Climate Day materials or supplemental / created by site): Brief Protocol for Interview with Earth Ambassador:  Provide general information on how the Earth Ambassador organized their Climate Day Event (e.g., structure of event, professionals contacted, publicity, etc.):  Were the events implemented as planned?  What successes and challenges did the EA experience as the Climate Day event was implemented? Workshop #1: ○ None of the time ○ A little ○ Sometimes ○ A lot ○ Throughout ○ N/A  How often did presenter focus on hands-on / interactive participant involvement?  Did the presenter encourage and value audience input (e.g., questions, comments)?  How often was lecture (as a teaching technique) used?  How often was breaking into smaller groups (as a teaching technique) used? ○ Not at all ○ A little ○ Somewhat ○ Very ○ Extremely ○ N/A  Was the science content appropriate for the developmental levels of the audience members?  Was presenter able to address audience questions? Some items adapted from Goodman Research Group, Inc. (2009). Final evaluation report: Community Ambassadors in Science Exploration (CASE). Retrieved from http://informalscience.org/reports/0000/0215/CASE_Final_Report_-_GRG__Inc..pdfhttp://informalscience.org/reports/0000/0215/CASE_Final_Report_-_GRG__Inc..pdf

13. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Survey for Kopernik Conference Participants To what extent did…[Presenter A] provide sufficient information regarding causes of extreme weather and climate change? What was one key "take away" from this conference for you? Thinking back to what you learned at the conference, what three or four things did you hope to accomplish when you returned to your community? What additional support would you need in order to take action—or additional action— related to minimizing the damage due to extreme weather and climate change in your community? Phone Interview Protocol for Kopernik Conference Participants In your survey responses, you indicated that since the conference you were able to X (actions described in survey responses). What challenges have you experienced as a result of these actions? What successes have you experienced? What type of reaction have you received from the community? From stakeholders? Do you have additional action plans related to due to mitigating greenhouse gases or minimizing the damage from extreme weather? What resources or information would you need in order to accomplish these plans?

14. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation

15. Perceived Importance of Community Engagement “Citizens have to be informed and want to adapt to the changes…The information has to be regularly supplied to the community, otherwise it is out of sight, out of mind.” “It’s very important to do this type of education and it’s valuable to have local people to know who and where to target…The local politicians are generally ignorant or just disregard it. We’re in a flood-prone area so it’s good to go after the people who are directly affected by it because then they have a greater understanding of the connection between climate and weather.” “If I were with NASA, I’d ask, how can we work more effectively with public schools and institutions? What can they influence? …make it in the best interest of research universities to see community-based education as a top priority for which they will receive money and recognition.” “Many people in U.S. aren’t very literate in solar energy but we can fix that. On-the-ground work is the most effective work, and I think the [EA program] is a great idea.”

16. Research ● Evaluation ● School Improvement ● Learning Innovation ● Educator Effectiveness ● Systems Transformation Thank you!!