1. EVALUATING A MIDDLE SCHOOL MATH M.ED. PROFESSIONAL DEVELOPMENT PROGRAM. Knowledge, Pedagogy, Practice or Student Achievement:

2. Program Description Need:  Data from LEA indicated many teachers in district teaching math, without math endorsement or math education Program design:  M.Ed. Program that focused on Middle School Math and included integration of science. Course in Geometry, Number Theory, Algebra, Probability and statistics, math methods, Research Methods, Action Research, and Practicum  Cohort program enrolling 30 candidates into two cohorts who progress through the program at the same time  Collaborative program between LEA, Loyola’s School of Education and Loyola’s college of Arts and Science  LEA would cover one third of tuition and Loyola University would cover other two thirds.

3. Evaluation Design Funding Agency Requirements:  Experimental or quasi-experimental design  Use prescribed data collection instruments  Emphasized standardized student achievement data as outcome for program's success Evaluator's Design  Emphasized quasi-experimental design using two cohorts as experimental and control groups for each other  Used nonequivalent control group design for student achievement data  Interviews and observation for qualitative data on practice  Emphasized changes in content knowledge, pedagogical perspectives and classroom practices as outcomes for program success.

4. Life Happened Program Changes  Low enrollment only one cohort group  LEA withdrew causing funding issues  Original program director stepped down  New Program director came on board Evaluation changes  Pretest posttest only design for content knowledge and pedagogical perspectives, and student achievement  Observations and interviews of classroom practice  Evaluator, Program Director, and Graduate student form evaluation team.  Agency emphasis still student achievement data, Evaluation team emphasized knowledge, pedagogy and practice.

5. Data Collection for Outcomes Student Achievement:  ISAT Scores for two consecutive years Math Content Knowledge:  DTAMS pre/post Algebraic Ideas, Geometry and Measurement, Number Computation, and Probability and Statistics Fact, conceptual understanding, problem solving and pedagogical knowledge scores Pedagogical Perspective:  Classroom Mathematics Teachers Questionnaire  Focus group Practice:  Inside the Classroom Observation and Analytic Protocol  Focus group

6. Analysis of Quantitative Outcome Data Student Achievement  ISAT data inconclusive on program’s success or value Content Knowledge  Significant increase in content knowledge for Algebra, Probability and Statistics, and number theory courses Geometry course results questionable Pedagogical Perspectives  Significant changes in several areas related to the the candidates perspectives of math instruction

7. Analysis of Qualitative Outcome Data Observations and Interviews Increase in content knowledge Increase in self-efficacy Changes in pedagogical perspectives of math instruction Changes in classroom practices Integration of Math and Science instruction Substantiated pedagogical questionnaire results Focus groups Substantiated observation and interview data results Provided information about student achievement Provide feedback on program improvement.

8. Determining Program Success: Distal vs. Proximal Outcome Measures Distal –  ISAT student achievement data  Confounding evidence of program success Confounded by third variables Limited amount of data Limited time for program to have impact on students Proximal –  DTAMS  Evidence of increase in candidates content knowledge as a result of course work  Pedagogical Questionnaire  Evidence of changes in pedagogical perspectives related to math as a result of course work

9. Determining Program Success: Distal vs. Proximal Outcomes Measures Proximal –  Observations and Interviews  Evidence of changes in candidates classroom practices as a result of course work  Evidence of changes in candidates pedagogical perspectives as result of course work  Focus group  Evidence of changes in candidates self-efficacy  Evidence of changes in candidates classroom practices as a result of course work  Evidence of changes in candidates pedagogical perspectives as result of course work  Suggests increase in student achievement

10. Conclusions Proximal outcomes provided a better means of measure the program successes Proximal outcomes supported logic model for program Proximal outcomes provided a better understanding of program value Emphasis on program value needs to be closely linked to the program purpose with an emphasis on change related to knowledge, pedagogy and practice.