Download presentation
Presentation is loading. Please wait.
Published byBrooke Parks Modified over 9 years ago
1
S-STEM Program Evaluation S-STEM PI Meeting Arlington, VA October 2012
2
Abt Associates | pg 2 Program Evaluation Overview NSF need for a comprehensive examination of implementation, effectiveness and impact of S-STEM Dual evaluation focus –Scholarship recipients and role of S-STEM in their educational opportunities and employment outcomes –Institutions and the student support they provide Evaluation team: Abt Associates, SageFox, Abt SRBI
3
Abt Associates | pg 3 Key Evaluation Features Collaborative relations with NSF, projects, and other experts Evaluation based in program theory Involves 494 projects –Funded from 2006 to 2010 –Involve undergraduate students
4
Abt Associates | pg 4 Challenges and Opportunities Challenges –Tracking scholarship recipients –Obtaining cooperation and high response rates –Identifying an appropriate comparison Opportunities –Collecting data representative of projects and program –Providing NSF with a strong understanding of investment –Contributing to understanding of student supports and success
5
Abt Associates | pg 5 Study Components Implementation study –Ways in which S-STEM projects recruit and retain students in STEM fields, allocate scholarship funds, and provide educational and support programming for scholarship recipients Relational study –Associations between project characteristics and practices and recipient outcomes Benchmarking comparison –Comparisons of recipients’ educational and academic support experiences to national trends Quasi-experimental study –Use of Propensity Score Matching (PSM) to compare the educational and career outcomes of S-STEM scholarship recipients to a matched national sample
6
Abt Associates | pg 6 Multiple Data Sources S-STEM Monitoring System –Components and strategies employed by S-STEM projects Projects’ Annual Reports –Descriptions of activities and implementation Site visits to a purposive sample of S-STEM projects –In-depth information about project implementation in particular contexts Survey data from S-STEM Principal Investigators –Operation and implementation of S-STEM awards Survey data from S-STEM scholarship recipients –Information on educational experiences, and r educational and career outcomes Beginning Postsecondary Students Longitudinal Study (BPS) survey –Extant data on outcomes from a matched comparison group
7
Abt Associates | pg 7 Use of Monitoring System Identify population –5,478 at two-year institutions –14,914 at four-year institutions Identify supports and activities (2-year n=74, 4-year n=350) –Academic support services (96%, 94%) –Participation in meetings or conferences (86%, 63%) –Career counseling (76%, 70%) –Mentoring by faculty or professionals (73%, 94%)
8
Abt Associates | pg 8 Site Visits to Selected Awards Emerging themes –Building on an existing structure –Developing a community of scholars –Providing talent for the local workforce –Recruiting through an existing program –Providing early access to research –Recruiting to the university –Strategic alignment with other university initiatives –Mentoring/advising –Creating pathways
9
Abt Associates | pg 9 PI Survey Topics S-STEM activities Recruitment and selection Scholarship recipients Recipient outcomes Benefits and challenges
10
Abt Associates | pg 10 Recipient Survey Topics Education background College experiences –Supports –Work –Financial aid Future plans Employment
11
Abt Associates | pg 11 Critical Role of PIs and Recipients Ensure high response rate on surveys –OMB threshold is 85% Cooperation with requests from Abt Associates –Complete surveys –Encourage participation Timeframe –NSF and OMB review required before fielding –Surveys in field Spring or Fall 2013
12
Abt Associates | pg 12 Measure Progress toward Goals Improved educational opportunities for students Increased retention of students to degree achievement Improved student support programs at institutions of higher education –Increased numbers of well-educated and skilled employees in technical area of national need
13
Abt Associates | pg 13 Uses of Evaluation Reporting progress and accomplishments to stakeholders –Congress, OMB, GAO, OSTP, NSF, the public Accountability –Numbers of STEM graduates, time to degree, retention, success of project activities Program improvement Inform similar types of programs
14
Abt Associates | pg 14 Questions alina_martinez@abtassoc.com
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.