1. Defeating the Many-Headed Monster: Strategies for Effective Comprehensive Program Evaluation Joseph B. Berger School of Education University of Massachusetts Amherst

2. Purpose To provide an overview of research designs and strategies that can be used in the evaluation of complex, multi-faceted NSF funded projects such as STEMTEC.

3. Multiple Challenges Institutions  1 University  4 Colleges  3 Community Colleges Stakeholders  NSF  PIs  Faculty  Administrators Years Goals

4. STEMTEC Goals 1.Establish a functional educational collaborative. 2. Redesign the science and math curricula on the campuses of the Collaborative to incorporate new pedagogies and establish mechanisms for supporting faculty in their course redesign. 3. Improve preparation of future K-12 teachers of mathematics and science.

5. STEMTEC Goals - continued 4. Recruit and retain promising students into the teaching profession, with special attention to underrepresented groups. 5. Develop a program to support new science and math teachers in their first year in the classroom. 6. Establish dissemination mechanisms. 7. Conduct strong programs of evaluation and assessment.

6. Research Strategies Quantitative  Surveys  Secondary Data Analysis Qualitative  Interviews and Focus Groups  Classroom Observations  Document Analysis

7. Data Sources Eight College Campuses Faculty Administrators Students Classrooms

8. Data Collection FacultyStudentsAdmini- strators New Teachers SurveyXXX XX InterviewXXXXXXXXX ObservationXXXX XXX DocumentsXXXXX ArchivalX

9. Methodological Strategies Use existing data and adapt existing instruments Create new instruments Be formative and summative Build internal and external networks Utilize the Core Adapt existing instruments Develop longitudinal data Construct evidence across multiple  Sources  Years

10. Additional Strategies Create Flexible Blueprints  Chronological  Task  Individual Develop a Team with Complimentary Skills Construct the Table of Contents First Communicate with Stakeholders

11. Additional Advice Be Flexible Take time to understand complexities Don’t reinvent the wheel Use Conceptual Frameworks Document everything Have fun!