Pilot STEM in OST Evaluation Preliminary Report March 2012 University of California, Irvine

Study Sample Selected 17 programs in 9 Regions participating Pilot Evaluation Study 17 programs in 9 Regions participating Pilot Evaluation Study Regions 1, 2, 3 4, 7, 8, 9, 10, 11 Regions 1, 2, 3 4, 7, 8, 9, 10, 11 Total of 65 sites recruited Total of 65 sites recruited 52 Elementary 52 Elementary 9 Middle School 9 Middle School 4 K-8 4 K-8 Criteria for Selecting Study Sample: Criteria for Selecting Study Sample: Range of STEM curriculum approaches Range of STEM curriculum approaches Range of student age groups (grades 3-12) Range of student age groups (grades 3-12) Diverse ethnic and socio-economic backgrounds, representative of students in the State of California Diverse ethnic and socio-economic backgrounds, representative of students in the State of California Internet access Internet access

Overview of Evaluation Activities: Selected Study Sites engage in the following activities: Administer pre (fall 2011) and post (spring 2012) online student surveys Administer online staff surveys at two time points: pre (fall 2011), and end-of-year (spring 2012) Document Weekly Stem Activities—to be recorded daily by site implementers on STEM Activity Documentation Forms Provide UC Irvine with copies of program schedule and lesson plans

Pilot Study Launch Fall 2011 ACTIVITIES TO DATE:  Selection of pilot study sample sites  Set up of staff pre-survey and student pre-survey online links  Communication with pilot study sites: 1. sent informing Program Liaison of selected sites and overview of evaluation activities to be carried out 2.Individual s sent to STEM Implementers at each study site including: 1)Individual staff IDs assigned & instructions distributed for staff surveys 2)Site IDs assigned & instructions distributed for student surveys 3)STEM Activity Documentation Form (electronic and hard copies) distributed with instructions & envelopes for returning forms to UC Irvine 3.Packages sent to each study site or program liaison with: 1)Hard copies of STEM Activity Documentation Forms and instructions 2)Prepaid and addressed envelopes completed forms to UC Irvine

Winter 2012 Evaluation Activities  Follow-up Communications & Reminders: 1.Program liaisons sent a list of study sites and corresponding staff IDs and site IDs Nov. 30/Dec. 1, Reminders sent to Program Liaisons & STEM Implementers Reminders to complete surveys and begin sending STEM Activity Documentation Forms Reminders to complete surveys and begin sending STEM Activity Documentation Forms Accounting of data collected from their program sites Accounting of data collected from their program sites All site and staff codes and survey instructions resent with All site and staff codes and survey instructions resent with  Ongoing contact with program Program Liaisons & STEM Implementers by phone and

Data Collected to Date—March staff pre-surveys have been completed 35 sites have completed 1,277 student pre-surveys  Preliminary results of Staff and Student Pre-Surveys summarized in slides that follow 103 STEM Activity Documentation forms received reporting on 310 Individual Stem Activities  Items reported by STEM implementers about each activity include: Date and Duration of Activity Date and Duration of Activity Name of activity Name of activity STEM content area addressed STEM content area addressed Number of students and grade level Number of students and grade level 4 point ratings of 4 point ratings of 1.Level of student Engagement 2.Level of challenge 3. Overall assessment of success of activity  Data are being entered in the data base for analysis and will be correlated with staff survey data and student survey site level data.

Summary STAFF Pre-Survey Data Data collected December 2011–March 2012

SUMMARY OF STAFF PRE-SURVEY RESULTS  90 staff surveys collected between November 2011 and March 2012  73% respondents female.  1/3 are between 18 and 25 years old.  72% are 35 or under.

S TAFF S URVEY —L EVEL OF E DUCATION A CHIEVED Nearly half (48%) have attended college or have an AA degree 42% have B.A. Degrees or higher

Staff Reflect the Diversity of Students and Communities Served

S TAFF P OSITION IN A FTERSCHOOL P ROGRAM  40% staff have less than one year in current position  33% have more than 5 years in position  27% have 1-3 years in position  40% staff have less than one year in current position  33% have more than 5 years in position  27% have 1-3 years in position

S TAFF E XPERIENCE IN OTHER S CHOOL S ETTINGS 62% have some experience as a classroom aide or TA  23% 1-5 years14% more than 5 years 37% reported having some classroom teacher experience  15% 1-5 years10.5% more than 5 years 15% have some school administrative staff experience 8% student support staff experience Few staff report any administrative experience (6%) Professional Experience n< 6 mo6-12 mo1-2 yrs2-5 yrs5-10 yrs10+ yrs School Administrator74 3%00 00 Student Support Staff74 1%03%1%03% Administrative Staff76 3% 5%4%00 Classroom Teacher79 2.5%9%4%11%2.5%8% Instructional Specialist75 4%3% 9%3% Classroom Aide, TA84 14%11% 12%6%8%

Staff Self-Identified Instructional Roles

Variety of STEM Activities Implemented Prompt: In your current position, do you implement any STEM activities with students? If so, please specify which types of activities (select all that apply):

Prior Experience & Time per Week Implementing STEM Prompt: In your current position, how much time per week do you spend implementing STEM activities with students?  22% spend NO time implementing STEM  Nearly half (49%) do at least 30 minutes to 2 hours of STEM per week  30 % have no prior experience implementing STEM at another program

Staff Meetings Around STEM 21 % report never discussing STEM at staff meetings 55% discuss STEM in staff meetings at least once a month to once a week n =90 Frequency of Staff Discussion of Program and STEM Issues

Compensation for Staff Meetings ( n = 89) $66% staff report being compensated for all meetings $10% for most meetings $7% for some meetings $15% receive no compensation for meetings  2% staff report having no meetings

Training and Support A little over half (52%) of staff surveyed have had at least 1-4 sessions of Stem Related Training in the past academic year 42 % have had NO Stem Training n = 90

S TAFF R ELATIONS WITH T EACHERS During the past academic year, how often have you discussed…  32% never speak with classroom teachers about STEM concepts taught in classroom

S TAFF R ELATIONS WITH P ARENTS n = 89  76% of staff report never holding STEM-Related events for parents  36% say they never speak with parents about STEM activities  76% of staff report never holding STEM-Related events for parents  36% say they never speak with parents about STEM activities

Staff Outcome Measures Beliefs & Sense of Competency nRangeMeanSDAlpha Staff Beliefs about STEM Staff Sense of Competency Two staff outcome measures:  5 point rating scale o strongly disagree, disagree, neither agree or disagree, agree, strongly agree [some reversed coded]  Staff beliefs about STEM in the afterschool program. Example Items I think the students enjoy doing STEM Activities In general, I think these students [in the afterschool program] are very capable of doing hands-on science activities I don’t think there is enough time at the program for students to learn much about STEM  Staff sense of competency implementing STEM in the afterschool program Example Items Example Items I have a strong background in at least one area of STEM I do not know enough about Science, Technology, Engineering and/or Mathematics to teach any of them well I feel confident about teaching Science, Technology, Engineering and/or Mathematics in the afterschool program

Summary STUDENT Pre-Survey Data Data collected December 2011–March 2012

Student Survey Sample 1,277 in Full Sample 51% Boys and 49% Girls Grade and Gender of Student Respondents

Time to Complete Survey 51% students overall report taking 15 minutes or less to complete survey 55% Middle School & 28% Elementary students report taking 9 minutes or less

Survey taking Experience Full sample : n = 1200  The majority of students find the surveys Easy to Read, Understand and Answer 83-85% state “mostly true” or “really true” 83-85% state “mostly true” or “really true”

STUDENT OUTCOMES Pre-Survey Results 1,277 surveys completed by students grades 3-8 Eight scales: 1.Work Habits 2.Misconduct 3.Social Competencies 4.Math Efficacy 5.Science Efficacy 6.Interest and Engagement in STEM 7.Future Outlook 8.Science Career Aspirations

Work Habits  Work Habits—mean of 6 items  Assessed on a 4-point scale (1 = not at all true, 4 = really true)  Sample items include: “I work well by myself” “I work well by myself” “I finish my work on time.” “I finish my work on time.” Full Sample n = 1,277 SD =.67 Alpha =.81

Misconduct  Misconduct—mean of 9 items  Assessed on a 4-point scale: (0 = never, 3 = more than once a week) Lower score means LESS misconduct Lower score means LESS misconduct  Sample items include: “I have gotten into a fight at school” “I have gotten into a fight at school” “I have taken something that belongs to someone else.” “I have taken something that belongs to someone else.”

Social Competencies  Social Competencies—mean of 7 items  Assessed on a 4-point scale: (1 = not at all true, 4 = really true)  Sample items include: “I work well with other kids” “I work well with other kids” “I can tell other kids what I think, even if they disagree with me.” “I can tell other kids what I think, even if they disagree with me.”

Math Efficacy  Math Efficacy—mean of 4 items  Assessed on a 4-point scale: (1 = not at all true, 4 = really true)  Sample items include: “I expect to do well in math” “I expect to do well in math” “I am interested in math.” “I am interested in math.”

Science Efficacy  Science Efficacy—mean of 4 items  Assessed on a 4-point scale: (1 = not at all true, 4 = really true)  Sample items include: “I expect to do well in science” “I expect to do well in science” “I am interested in science.” “I am interested in science.”

Excited, Engaged and Interested Science Learner [PEAR-Harvard]  Excited, Engaged & Interested Science Learner—mean of 24 items  Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree) Some items reverse coded Some items reverse coded  Sample items include: “Science is something I get excited about.” “Science is something I get excited about.” “I am curious to learn more about science, computers or technology.” “I am curious to learn more about science, computers or technology.” “Science is boring” “Science is boring” “I pay attention when people talk about recycling to protect our environment.” “I pay attention when people talk about recycling to protect our environment.” “I enjoy visiting science museums or zoos.” “I enjoy visiting science museums or zoos.”

Future Outlook  Future Outlook—mean of 7 items  Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)  Sample items include: “I will go to college.” “I will go to college.” “I will have a job that I enjoy doing.” “I will have a job that I enjoy doing.”

Science Career Aspirations  Science Career Aspirations—mean of 4 items  Assessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)  Sample items include: “I will have a career in Science, Technology, Engineering, or Mathematics.” “I will have a career in Science, Technology, Engineering, or Mathematics.” “I will graduate with a college degree in a major area needed for a career in science.” “I will graduate with a college degree in a major area needed for a career in science.”

Next Steps Initiate Post-Survey administration: April 15, 2012-June 15, 2012 Ensure staff have received information, instructions and forms for completing Activity Documentation Forms UC Irvine to Request sample copies of site schedule and weekly lesson plan Final analysis of pre/post student and staff outcome measures and program implementation data Refinement of measures for STEM in OST evaluation study