Using Formative and Summative Evaluation to Improve Field Trip Programs Kimberly M. Burtnyk Amgen Center for Science Learning California Science Center

n Pre-/Post-visit classroom activities Curriculum compatible K8 CA standards n Gallery worksheets (atypical) “Chaperone Sheets” è Available free from our website ThinkSCIENCE! Pathways Components

n Help teachers implement curriculum- compatible field trips n Increase educational guidance for field trips n Increase interactions with staff  Help chaperones facilitate  Increase concept understanding through chaperone facilitation ThinkSCIENCE! Pathways Goals of the program

Formative Evaluation Spring 2002 Focus: Access, Use, Chaperone/student behavior Summative Evaluation Timeline Spring 2003 Focus: Cognitive gains ThinkSCIENCE! Pathways Evaluation Plan

n Recruiting Groups with existing reservations n Questionnaires Pathways usage issues Field trip planning and behavior Chaperone roles n Observations Stopped/didn’t stop Time spent--we want Pathways groups to stay longer Panels Read, Discussions, Gestures Formative Evaluation Method

n Recruiting 37 Schools recruited from March-June n Questionnaires 82 Returned (37 Teacher 45 Chaperone) n Observations 15 schools tagged for observation (8 PW, 7 Non-PW) 72 Chaperones observed Formative Evaluation Results Summary

n Only 3 of 8 PW recruits used it n Reasons  Technical difficulties Materials looked too complicated and long Confused about where to find them  Unaware of the materials Timing Formative Evaluation Questionnaire Highlights

72 Chaperones observed Formative Evaluation Observations Highlights Stopped/Did not Stop at Capsules

Mean t: 66 s Mean t: 138 s Formative Evaluation Observations Highlights Time spent at Capsules

Of the 35 chaperones who stopped... Formative Evaluation Observations Highlights Reading, Discussion, Gestures

n Pathways usage was limited due to: Access to computers/printers Length and perceived complexity of the materials  Communication gaps between school administrators and teachers BUT... it did produce some significant behavioral changes in chaperones who used it Formative Evaluation Conclusions

How Did Formative Evaluation Impact the Program? n In-class/On-site activities separated n Shortened Map removed One activity removed (Gallery incomplete) n More direct link created quick easy obvious

Summative Evaluation Method n Recruiting Talked directly to lead teacher Materials sent to groups In-class visit prior to field trip n Pre/Post Test n Questionnaires--dropped n Observations Gestures dropped

Summative Evaluation Results Summary n 298 Students Recruited 217 had in-class visit n 254 Pre- and Post-tests completed 28 PW 226 Non-PW n Observations 26 chaperones observed  ONLY 4 CHAPERONES USED PATHWAYS

Summative Evaluation Conclusions n Jury still out on cognitive impact of Pathways The numbers are too small Will resume attempts next year n Pre/Post test shows impact of In-class Visit Micrometeoroid question Chi Square p<0.004 for those who had visit vs those who did not

1. Even with intensive intervention and support, teachers did not use PW  There were special circumstances in the A&S Gallery  And timing, curriculum, age groups were an issue too BUT:  Are such materials worth spending $$ developing?  What do teachers really want/need, if anything? We don’t yet know. 2. Whether or not PW improves learning outcomes as the lit. suggests is still unknown  Warrants further testing and refinement of good and USEFUL field trip practices. What did Formative and Summative Evaluation Teach us About Pathways?

3. We really need to learn more about teachers and chaperones  Can’t assume that teachers have access to net sources  Chaperones might be willing facilitators 4. We can apply statistics to tell us about the use and effectiveness of the program  Further refining statistical techniques  Learning curve  Exciting prospects What did Formative and Summative Evaluation Teach us About Pathways?