S-STEM Program Evaluation S-STEM PI Meeting Arlington, VA October 2012

Abt Associates | pg 2 Program Evaluation Overview  NSF need for a comprehensive examination of implementation, effectiveness and impact of S-STEM  Dual evaluation focus –Scholarship recipients and role of S-STEM in their educational opportunities and employment outcomes –Institutions and the student support they provide  Evaluation team: Abt Associates, SageFox, Abt SRBI

Abt Associates | pg 3 Key Evaluation Features  Collaborative relations with NSF, projects, and other experts  Evaluation based in program theory  Involves 494 projects –Funded from 2006 to 2010 –Involve undergraduate students

Abt Associates | pg 4 Challenges and Opportunities  Challenges –Tracking scholarship recipients –Obtaining cooperation and high response rates –Identifying an appropriate comparison  Opportunities –Collecting data representative of projects and program –Providing NSF with a strong understanding of investment –Contributing to understanding of student supports and success

Abt Associates | pg 5 Study Components  Implementation study –Ways in which S-STEM projects recruit and retain students in STEM fields, allocate scholarship funds, and provide educational and support programming for scholarship recipients  Relational study –Associations between project characteristics and practices and recipient outcomes  Benchmarking comparison –Comparisons of recipients’ educational and academic support experiences to national trends  Quasi-experimental study –Use of Propensity Score Matching (PSM) to compare the educational and career outcomes of S-STEM scholarship recipients to a matched national sample

Abt Associates | pg 6 Multiple Data Sources  S-STEM Monitoring System –Components and strategies employed by S-STEM projects  Projects’ Annual Reports –Descriptions of activities and implementation  Site visits to a purposive sample of S-STEM projects –In-depth information about project implementation in particular contexts  Survey data from S-STEM Principal Investigators –Operation and implementation of S-STEM awards  Survey data from S-STEM scholarship recipients –Information on educational experiences, and r educational and career outcomes  Beginning Postsecondary Students Longitudinal Study (BPS) survey –Extant data on outcomes from a matched comparison group

Abt Associates | pg 7 Use of Monitoring System  Identify population –5,478 at two-year institutions –14,914 at four-year institutions  Identify supports and activities (2-year n=74, 4-year n=350) –Academic support services (96%, 94%) –Participation in meetings or conferences (86%, 63%) –Career counseling (76%, 70%) –Mentoring by faculty or professionals (73%, 94%)

Abt Associates | pg 8 Site Visits to Selected Awards  Emerging themes –Building on an existing structure –Developing a community of scholars –Providing talent for the local workforce –Recruiting through an existing program –Providing early access to research –Recruiting to the university –Strategic alignment with other university initiatives –Mentoring/advising –Creating pathways

Abt Associates | pg 9 PI Survey Topics  S-STEM activities  Recruitment and selection  Scholarship recipients  Recipient outcomes  Benefits and challenges

Abt Associates | pg 10 Recipient Survey Topics  Education background  College experiences –Supports –Work –Financial aid  Future plans  Employment

Abt Associates | pg 11 Critical Role of PIs and Recipients  Ensure high response rate on surveys –OMB threshold is 85%  Cooperation with requests from Abt Associates –Complete surveys –Encourage participation  Timeframe –NSF and OMB review required before fielding –Surveys in field Spring or Fall 2013

Abt Associates | pg 12 Measure Progress toward Goals  Improved educational opportunities for students  Increased retention of students to degree achievement  Improved student support programs at institutions of higher education –Increased numbers of well-educated and skilled employees in technical area of national need

Abt Associates | pg 13 Uses of Evaluation  Reporting progress and accomplishments to stakeholders –Congress, OMB, GAO, OSTP, NSF, the public  Accountability –Numbers of STEM graduates, time to degree, retention, success of project activities  Program improvement  Inform similar types of programs

Abt Associates | pg 14 Questions