Differential Effects of Participatory Evaluation in a National Multi-site Program Evaluation Frances Lawrenz University of Minnesota

Collaboratives for Excellence in Teacher Preparation CETP program promotes comprehensive change in the undergraduate education of future teachers by supporting cooperative, multiyear efforts to increase substantially the quality and number of teachers well-prepared in science and mathematics, especially members of traditionally underrepresented groups.

CETP Characteristics Each CETP was unique All engaged faculty from STEM and education All included several institutions of higher education (including community colleges) in a particular geographic area All included mechanisms for improving undergraduate education in the sciences and mathematics. All had some sort of relationship with schools within the geographic area. All offered scholarships to students from underrepresented groups. The ranges within these were broad however e.g., from 3 institutions to 13 or more, and all had unique elements like liaisons with Tribal Colleges or industrial/science internship placements, etc.

History of CETP Evaluation Initially, project level emphasis on evaluation Program monitoring evaluation required—negative experience First conference with evaluators— Corridors for Collaboration Second conference with evaluators Funding of CETP CORE evaluation

Visit our web site

CORE Evaluation Questions Three main areas: 1. Institution 2. K-12 Teachers 3. K-12 Students Foci: 1. Collaborations 2. Comparisons to standards 3. Comparisons to other groups

Institutions a. How supportive of SMET reform education policies and procedures are the participating CETP institutions? SOURCES: Deans/Department Chairs, Faculty, PI b. How successful have the CETPs been at course reform? SOURCES: Deans/Department Chairs, Faculty, College Students, PI c. What impact has CETP had on the system or structure of the teacher education systems at the participating institutions? SOURCES: Deans/Department Chairs, Teachers, Principals, PI, Faculty, QRC

Teachers a. How well do CETP teachers demonstrate the knowledge and skills espoused by the National Science, Mathematics and Technology Education standards? SOURCES: K-12 Students, CETP Teachers, Classroom Observation Protocol, Rubric, Principals of CETP b. How do CETP teachers and the classrooms they create differ from non-CETP teachers and the classrooms they create? SOURCES: K-12 Students, Teachers, Classroom Observation Protocol, Rubric, Principals c. What outcomes have the participating higher education institutions, their faculty members, or the CETP contributed to the K-12 schools? SOURCES: CETP Teachers, Deans/Department Chairs, Principals, PI, Faculty

Students a. Are students learning what is expected in the SMET education standards? SOURCES: K-12 Students, PI, Classroom Observation Protocol, Rubric b. Are there differences in student outcomes for CETP and non-CETP teachers? SOURCES: K-12 Students, Classroom Observation Protocol, PI

Positives Good working relationships Shared expertise; website, list serve, s, and meetings Quality data collection instruments Group negotiations with NSF More effort on unique aspects Incentives; $15,000, print and mail out instruments, help with IRB, data entry and return

Negatives Instruments not matched exactly to CETP Time and labor intensive data collection All levels of participants required to participate Training required to do classroom observations Not required by NSF

Recommendations Plan for the program evaluation before funding begins Require participation in the evaluation Consider partial participation—either only some sites participating or some sites collecting only some data Have project evaluators in the loop from the beginning Substantively involve PIs Have the program evaluation data play a critical role in project evaluation