1. Evaluating NSF Programs
Dr. Jennifer Giancola Carney, Abt Associates
September 18, 2008

2. Agenda Two NSF program evaluations (IGERT & CAREER) Q&A discussion
Design & findings
Rationale for methods used
Limitations of methods used
Lessons learned
Q&A discussion
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

3. Since 1998, PhD training program (DGE)
Integrative Graduate Education and Research Traineeships (IGERT) Program
Since 1998, PhD training program (DGE)
Grants to universities who develop new IGERT-related programs (most $$  student traineeships)
Give PhD students interdisciplinary research experiences and enhanced professional skills & perspectives
Three phases of evaluation
Implementation study ( )
Impact study ( )
Follow-up study of graduates (2006-present)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

4. IGERT Evaluation: Began with a Logic Model
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

5. IGERT Implementation Study
Annual Monitoring: “Who? What? When?” research questions
Who participates and why? What activities are conducted?
Annual web survey of program participants (PIs & trainees)
Describe the program recruitment strategies, training activities, faculty involvement
Site Visits: “How? Why?” research questions
What challenges have projects encountered? How have they overcome them?
Interviews with faculty, students, chairs, administrators
Identify common challenges and solutions project management, faculty engagement, implementing interdisciplinary education within universities
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

6. IGERT Implementation Study (cont.)
Examine implementation across projects and over time
Mixed methods (quantitative & qualitative data)
Data used for GPRA reporting, program management, revisions to solicitations, sharing common solutions with IGERT PIs
Limitations
Little information on longer-term effects of IGERT or broader program impacts on faculty and the university
No comparison to non-IGERT experiences to take into account overall trends in graduate education
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

7. 2003 IGERT Impact Study Impact study: So what?” research questions
What have been the outcomes for participating IGERT faculty and students as compared to non-participating faculty and students?
Has there been any institutional impact of IGERT funding?
IGERT participants (PIs, dept chairs, faculty, students, administrators) and Non-IGERT participants (comparison group)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

8. IGERT Comparison Group
Provide a counterfactual for what would have been had IGERT not existed. Needs to control for “academic quality” and variations among STEM disciplines
Matched each IGERT department to a non-IGERT department with whom they compete for graduate students
Vulnerable to selection bias: Outcomes may be due to pre-existing characteristics of IGERT students, not to IGERT program
Examples of reported findings:
Can say: “IGERT trainees engage in more interdisciplinary activities as graduate students than non-IGERT students.”
Cannot say: “IGERT causes students to engage in more interdisciplinary activities.” (Maybe these students would have sought out i/d activities regardless.)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

9. IGERT Impact Study (cont.)
Benefits
Examine value of IGERT for:
Departmental recruitment
Student preparation
Faculty interdisciplinary involvement
Institutional offerings and support for interdisciplinary education
Assess against counterfactual of “traditional” graduate ed.
Limitations
Focused on current participants
Tested lots of outcomes – hypothesis generating (not confirming)
No data on longer term outcomes for graduates
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

10. 2006 IGERT Follow-up Study
Graduate study: “What?” “So what?” questions:
Where do IGERT graduates go and what do they do? Are they any different from non-IGERT graduates? Has IGERT helped prepare them for their chosen careers?
IGERT graduates and comparison group of non-IGERT graduates
Presenting detailed descriptive data on IGERT graduates
Limiting outcomes tested with comparison group to key outcomes (hypothesis confirming, though still selection bias)
Challenge: locating graduates
Monitoring system had info on point of contact
Easier to find those in academic positions versus non-academic positions. Introduces sample bias into results – will conduct non-response bias during analysis.
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

11. Faculty Early Career Development (CAREER) Program
NSF’s primary support mechanism for junior faculty members since 1995.
Grants to individual faculty members
Support the research and early career advancement of junior researchers
Promote the integration of research and education:
Individual awardees
Changing university culture
Most recent evaluation:
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

12. CAREER: Research Questions
Descriptive questions re: perceptions of CAREER
How do stakeholders at NSF perceive the CAREER program and its relationship to the mission of NSF?
How do faculty members in departments that host CAREER awardee(s) view the CAREER program and its relationship to their research and educational missions?
Impact questions
What is the impact of CAREER on the research activities and career advancement of awardees?
What is the impact of CAREER on the integration of research and education by faculty members?
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

13. CAREER: Methodology Descriptive Study
Interviews with NSF Program Officers
Survey of 700 department chairs
Site visits 22 departments
Samples representative of population in question (but no comparison group)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

14. CAREER: Methodology (cont.)
Quasi-experimental evaluation of impact on awardees
CAREER Awardees
Comparison group of Non-Awardees (same research potential and interest in integration of research and education)
Unsuccessful CAREER applicants who won another NSF grant as PI w/in 5 years of CAREER application
Matched using propensity scores (reduces selection bias)
Limited outcomes tested (confirming hypotheses)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

15. CAREER: Findings
Description of how program goals are interpreted within and outside of NSF  Inform program management
Description of characteristics of awarded PIs  NSF program reporting (GPRA, etc.)
Assessment of grant’s impact on awardees (“Receipt of a CAREER award increases the likelihood of receiving tenure”)  Inform decisions about program continuation or modification
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

16. Lessons learned
Know thy program: Until you understand the intervention, you cannot assess outcomes
Logic model, articulate goals
Develop indicators / measures of program success
Clearly define your research questions
Prioritize - you cannot evaluate everything
Identify data needs for reporting, decision-making
Be realistic (ask questions that can be answered about indicators that can be measured)
CAREER: “impact on institutional culture”
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

17. Lessons learned (cont.)
Identifying appropriate comparison groups
What’s the right counterfactual? Each comparison option allows you to answer different questions. Choose the option which best addresses the research questions.
IGERT - other interdisciplinary programs? Same or different institutions? All STEM students nationwide?
Choose right level of rigor (developing or testing hypotheses?)
Consider risk of selection bias
Change over time (longitudinal studies; pre/post)
Take advantage of available data available
National datasets
CAREER – data available to do PS matching
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

18. Lessons learned (cont.)
Ground each subsequent phase in findings from previous work:
Work from exploratory / descriptive evaluation to more summative / confirmatory evaluation. Each phase can answer questions raised (or not answered) in previous phases.
IGERT: Implementation  Impact  Graduate Follow-up
Take advantage of different levels of data collection
Qualitative versus quantitative; single versus cross-site
IGERT: Richness of single site visits enabled future studies
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

19. Lessons learned (cont.)
Think long term
Begin evaluation when program begins
Plan now for information you will need in the future
IGERT: tracking graduates
New study (GK-12) – building comparison group today for work in future
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008

20. Questions? Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008