1. © 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26, 2010 By the Center for Education Policy Dr. Raymond McGhee and Dr. Nancy Adelman

2. © 2007 SRI International2 Presentation Goals Provide background on program evaluation, leadership and team Describe purposes of the evaluation Describe each component of program evaluation and how it affects the projects and their leadership Share lessons learned from evaluation design and data collection so far Answer questions to clarify any aspect of the evaluation

3. © 2007 SRI International3 Evaluation Leadership and Experience Contracted by NSF to serve as an outside, third party evaluation contractor Nancy Adelman – Project Director Ray McGhee – Deputy Director Team of twelve researchers and analysts Experience in large-scale program evaluations

4. © 2007 SRI International4 Program Evaluation Purposes Describe and document program implementation Assess how the program is promoting change in undergraduate computing Support NSF in it’s reporting to the Academic Competitiveness Council.

5. © 2007 SRI International5 Project Evaluation Purposes Provide formative feedback to facilitate project refinements Describe initial implementation activities, successes, and challenges Monitor status of project activities Collect evidence of project successes and outcomes

6. © 2007 SRI International6 Relation between Program and Project Evaluation Data from projects will be incorporated into the program evaluation monitoring efforts Data from Special project evaluators will provide data to be used to document the program’s impact Data from the program evaluation will be reported in the aggregate and disseminated to all the projects

7. © 2007 SRI International7 Evaluation Data Sources 1. Monitoring Survey Data on project implementation for all projects Data will be used by NSF to monitor and track project activities Data analyses will be made available to projects in aggregate Timeline of first data collection: Feb 2010 – mid April 2010 Analysis & reporting: late May 2010

8. © 2007 SRI International8 Evaluation Data Sources 2. Site Visits Data for a cross-section of 10-12 projects in alternating years to document project implementation in context from the participants’ perspectives Purposive sample with case selection criteria: project type, cohort year, location, institution type Data will be analyzed across projects and reported in the aggregate to NSF and projects. 2010 Timeline: April – May (some in summer)

9. © 2007 SRI International9 Evaluation Data Sources 3. Analysis of Special Evaluations America Competes Act (HR 2272) mandates rigorous designs for evaluating STEM education programs SRI is supporting 24 projects conducting evaluations using quasi- experimental designs and will incorporate into analysis Coordination and communication between evaluator and PI/project team will be important Timeline: April – August 2010

10. © 2007 SRI International10 Evaluation Data Sources 4. Faculty Survey Sampling faculty from all projects Faculty attitudes and practices within departments promoting change in computing Survey development and piloting will occur in fall 2010 Data will be presented to NSF and projects in the aggregate Timeline: February – April 2011 & 2013

11. © 2007 SRI International11 Additional Resources from SRI CPATH Monitor Web site Hosting meetings of the special evaluators to support quasi-experimental designs Assistance in completing monitoring survey

12. © 2007 SRI International12 Lessons Learned The evolving nature of computing education reform and the diverse reform activities being implemented The role of institutional and regional context Avoiding the premature assessment of program impact Growing interest among different stakeholders Shared interests and benefits related to evaluation (for the CPATH program and projects)

13. © 2007 SRI International13 Summary of Next Steps in Evaluation Finalizing monitoring survey data collection and analysis Conducting site visits and analyzing data Meeting with special evaluators and collecting impact data Faculty survey development, pilot, clearance, and administration Annual reporting to NSF and ACC

14. © 2007 SRI International14 Questions & Answers Wrap-Up