1. MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09

2. TRC Criteria How has your Regional Collaborative impacted Teacher Content Knowledge Classroom Practice Student Achievement How has your Regional Collaborative impacted Teacher Content Knowledge Classroom Practice Student Achievement

3. Guskey Model Based on mediators of relationship between professional development and improvements in student learning i.e. administrator knowledge and practice, school culture, teacher knowledge and practice, parent knowledge and practice Based on mediators of relationship between professional development and improvements in student learning i.e. administrator knowledge and practice, school culture, teacher knowledge and practice, parent knowledge and practice

4. Five Critical Levels of PD Evaluation Inform planning, formative and summative evaluations Helps to identify the “what” and the “why” evaluation Hierarchically arranged from simple to complex Success at one level is necessary for success at following levels Inform planning, formative and summative evaluations Helps to identify the “what” and the “why” evaluation Hierarchically arranged from simple to complex Success at one level is necessary for success at following levels

5. Level 1: Participants’ Reactions Did they like it? Will it be useful Was the leader knowledgeable and helpful? Where the facilities comfortable? Did they like it? Will it be useful Was the leader knowledgeable and helpful? Where the facilities comfortable? Questionnaires Interviews Participant journals What questions are addressed?How will information be gathered?

6. Level 2: Participants’ Learning Did participants acquire the targeted knowledge and skills? Questionnaires Interviews Participant journals/portfolios Demonstrations What questions are addressed?How will information be gathered?

7. Level 3: Organization Support and Change Did it affect organizational climate, procedures, or policies Were sufficient resources made available? Were successes recognized & shared? Did it affect organizational climate, procedures, or policies Were sufficient resources made available? Were successes recognized & shared? Questionnaires Structured Interviews Participant journals/portfolios Focus groups District/school records What questions are addressed?How will information be gathered?

8. Level 4: Participants’ use of new knowledge and skills Did participants effectively apply the new knowledge and skills? Questionnaires Structured Interviews Participant journals/portfolios/ reflections Direct observation Videotapes What questions are addressed?How will information be gathered?

9. Level 5: Student learning outcomes Did it affect student performance or achievement? Are students more engaged learners? Do students select more rigorous coursework? Did it affect student attitudes toward science or math courses/careers? Did it affect student performance or achievement? Are students more engaged learners? Do students select more rigorous coursework? Did it affect student attitudes toward science or math courses/careers? Questionnaires School/student records Structured Interviews Participant journals/portfolios/ reflections Direct observation Videotapes What questions are addressed?How will information be gathered?

10. NSF Developed Tools Horizon Research Assessing Teacher Learning About Science Teaching (ATLAST) www.horizon-research.com/atlast/ Instruments to measure student and teacher science content knowledge Force and Motion Flow of Matter and Energy in Living Systems Plate Tectonics Horizon Research Assessing Teacher Learning About Science Teaching (ATLAST) www.horizon-research.com/atlast/ Instruments to measure student and teacher science content knowledge Force and Motion Flow of Matter and Energy in Living Systems Plate Tectonics

11. ATLAST continued Student tests - paper copies only Teacher tests - paper or online versions The student and teachers assessments measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts. Student tests - paper copies only Teacher tests - paper or online versions The student and teachers assessments measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts.

12. ATLAST continued HRI provides the following analyses: Percent of respondents choosing each answer choice; Percent of respondents answering each item correctly; Group mean score; and Significance testing and computation of effect sizes for pre- and post-test group mean scores. HRI provides the following analyses: Percent of respondents choosing each answer choice; Percent of respondents answering each item correctly; Group mean score; and Significance testing and computation of effect sizes for pre- and post-test group mean scores.

14. MOSART Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART) Harvard; PI - Phillip Sadler Content instrument for K-12 physical science and earth science, based on research about student misconceptions Completion of online tutorial (~30 min) required to download instruments Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART) Harvard; PI - Phillip Sadler Content instrument for K-12 physical science and earth science, based on research about student misconceptions Completion of online tutorial (~30 min) required to download instruments www.cfa.harvard.edu/smgphp/mosart

16. MOSART continued K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science Multiple versions of the test are available for pre/post administration K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science Multiple versions of the test are available for pre/post administration

17. MSPnet Toolbox http://hub.mspnet.org/index.cfm/msp_tools Clearinghouse for tools used and shared by MSP projects Tools may include assessment instruments, evaluation protocols, form letters, etc. http://hub.mspnet.org/index.cfm/msp_tools Clearinghouse for tools used and shared by MSP projects Tools may include assessment instruments, evaluation protocols, form letters, etc.

19. University of Louisville Diagnostic Assessments for Middle School Teachers Each assessment is composed of 25 items—20 multiple-choice and 5 open- response. Six versions of each assessment are available in paper-and-pencil format Diagnostic Assessments for Middle School Teachers Each assessment is composed of 25 items—20 multiple-choice and 5 open- response. Six versions of each assessment are available in paper-and-pencil format http://louisville.edu/education/research/centers/crmstd/diag _sci_assess_middle_teachers.html

20. University of Louisville (cont.) Assessments available free of charge Optional scoring service for $10/teacher Detailed summary of teachers' performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge) Assessments available free of charge Optional scoring service for $10/teacher Detailed summary of teachers' performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)

21. University of Louisville (cont.) Physical Science Life ScienceEarth/Space Science MatterStructure/FunctionAtmosphere/Hydrosphere Motion & ForcesInternal RegulationLithosphere EnergyHeredity/DiversitySpace Interdependence

22. MSP Knowledge Management and Dissemination Knowledge reviews good for citing research base Searchable Instrument database for science and math Knowledge reviews good for citing research base Searchable Instrument database for science and math http://www.mspkmd.net/

24. Online Evaluation Resource Library (OERL) http://oerl.sri.com/ http://oerl.sri.com/ Online Evaluation Resource Library (OERL) http://oerl.sri.com/ http://oerl.sri.com/ Teacher and student instruments Content assessments, interviews, surveys,classroom observations Teacher and student instruments Content assessments, interviews, surveys,classroom observations

26. Thinking About Mathematics Instruction (TMI) http://www2.edc.org/tmi/tmi_survey.html Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics http://www2.edc.org/tmi/tmi_survey.html Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics

28. Distributed Leadership http://www.sesp.northwestern.edu/dls/instruments/ Instructional Leadership Daily Practices Log Principal Experience Sampling Method (ESM) Log School Staff Network Survey http://www.sesp.northwestern.edu/dls/instruments/ Instructional Leadership Daily Practices Log Principal Experience Sampling Method (ESM) Log School Staff Network Survey