MSP PROJECT EVALUATION Lessons Learned Margaret Phelps & Kenneth W. Hunter, Sr. January 8, 2008 Millard Oakley STEM Center

Our MSP History Upper Cumberland Middle Grades Math Partnership ( ) TTU Engineering (7-12) Math Science Partnership ( ) Upper Cumberland Middle Grades Math Partnership ( ) Upper Cumberland Middle Grades Science Research Partnership ( ) Tennessee Pre-Engineering (7-12) Math Science Research Partnership ( )

Formative Assessment Participant Surveys (Institutes & Workshops) –Content –Format –Faculty –Scheduling –Facilities & food Basis for change/ improvement

Formative Assessment Daily faculty meetings during summer institutes –Observations of master teachers That was great! The teachers loved it! Don’t ever do that again! –Open discussion of issues Extremely helpful for “non-education” faculty Basis for “quick fix” improvements

Evaluation Designs Experimental Design with Random Assignment –LEAs provide list of eligible teachers (or teams) –Local evaluator does random selection Non-matched Comparison Groups –LEAs select teachers for intervention and control groups –Groups are compared statistically on base line measures to determine equivalence Descriptive--No comparison group

Evaluation Components Teachers –STEM Content Knowledge –STEM Pedagogy & Beliefs –Classroom Performance Students –STEM Content Knowledge –STEM Authentic Assessment

Teacher Content--Praxis II Advantages –External measure –Valid & Reliable –Tracks change over time Disadvantages –Cost –Obtaining score reports –Inflexibility of schedule

Instructor-made Test Advantages –Matches institute content –Aligns with standards –Control of testing Disadvantages –Not necessarily valid or reliable –Not normed –Forms might not be equivalent

Purchased Test Advantages –Control of testing –Access to scores Disadvantages –Cost –Single form –Alignment

Teacher Pedagogy & Beliefs Teacher-developed lesson modules Reports on lesson implementation Periodic self-assessment Surveys –Beliefs and Attitudes (Math) –Misconceptions (Science)

Classroom Performance Project-specific Observations –Conducted by LEA liaison or principal –Informs local leadership –Creates buy-in –Not valid for research purposes Research-based Observations –Requires training of observers –Cost (perhaps) –Useful for research purposes

Student Content Knowledge Middle school math & science –State testing program –Password-protected data available (can be disaggregated for analysis) –Compare NCE and proficiency level of students for pervious year (pre-) to year with project teacher (post-)

Student Content Knowledge High school math & science –8th grade scores are only common pre-test –HS scores not reported in NCE –Proficiency levels do not provide basis for strong evaluation –Many courses not tested –ACT only available for graduates

Student Content Knowledge Engineering Principles –JETS Assess (Junior Engineer Technical Society) –Will be administered in classrooms –Fall/Spring for comparison –Control of data collection –Will also be administered to teachers

Student Authentic Assessment Electronic Portfolio Based on Engineering Principles Multi-year – leads to pre-engineering certificate Curricular components –Project Lead the Way –Math, Science & Career Technical courses & projects

Student Authentic Assessment Electronic Portfolio Co-curricular components –FIRST LEGO League, FIRST Robotics –4-H and Scouting projects –Work experience –Service learning –Utilize local opportunities

Contact Information Dr. Margaret Phelps – –All TTU-related MSP programs & Millard Oakley STEM Center Mr. Kenneth W. Hunter, Sr. – –Tennessee Pre-Engineering Math Science Research Partnership ( ) Dr. Susan Gore – –Upper Cumberland Middle Grades Science Research Partnership ( ) Dr. Sandra Koczwara – –Upper Cumberland Middle Grades Math Partnership ( ) –Upper Cumberland Middle Grades Math Partnership ( ) Dr. Roy Loutzenheiser – –TTU Engineering Math Science Partnership ( )

The instructional practices and assessments discussed or shown in these presentations are not intended as an endorsement by the U.S. Department of Education.