Measuring Changes in Teachers’ Science Content Knowledge Dr. Anne D’Agostino Compass Consulting Group, LLC

2 Overview of Session Introductions Importance of Measuring Teacher Content Knowledge Measuring Teacher Content Knowledge Discussion

3 Importance of Measuring Teacher Content Knowledge US ED MSP Program Office Perspective The Government Performance Results Act (GPRA)  A law passed in 1993 that requires federally funded agencies to develop and implement an accountability system based on performance measurement. Importance of Measuring Teacher Content Knowledge

4 Requires all federally funded program to: Outline long-term and annual performance goals that include outcomes, Develop indicators to assess performance goals, Collect and analyze data on the indicators, and Report progress toward achieving performance goals based on the data collected and analyzed. MSP GPRA indicator related to content knowledge  The percentage of MSP teachers who significantly increase their content knowledge, as reflected in project-level pre- and post-assessment Importance of Measuring Teacher Content Knowledge

5 GPRA and PART are used by congress to inform appropriations – they want concrete evidence of results More uniform and higher quality data will enable the MSP Program office to more accurately and convincingly assess progress toward MSP performance goals This is challenging for measuring changes in teachers’ content knowledge Importance of Measuring Teacher Content Knowledge

6 MSP Grantee Perspective Has the potential to affect funding of the program at federal level US ED annual reporting requirement One of the more direct variables for individual partnerships to measure and demonstrate program outcomes Importance of Measuring Teacher Content Knowledge

7 Because of the variability in the activities projects are implementing to improve teacher content knowledge the use of one instrument is not a viable solution Some instruments are better than others Search for or develop an instrument that:  Is aligned with the content of your MSP professional development activity;  Has been piloted with teachers similar to those participating in your program; Measuring Teacher Content Knowledge

8  Provides evidence of reliability or the extent to which an instrument yields consistent, stable, and uniform results over repeated administrations under the same conditions each time  Provides evidence of validity or the extent to which 1) the instrument measures the skills it sets out to measure and 2) the inferences and actions made on the basis of the scores are appropriate and accurate Measuring Teacher Content Knowledge Figure obtained from the website: Discussion Question: What instruments have grantees been using? What are the properties of those instruments?

9 Potential Tests Assessing Teacher Learning About Science Teaching (ATLAST) - Horizon Research, Inc.  Three instruments developed in the following content areas to assess changes in teacher content knowledge: Force and motion, plate techtonics, and flow of matter and energy in living systems  Diagnostic Teacher Assessments in Mathematics and Science (D- TAMS) - University of Louisville  Three pre-post assessments of middle school science teacher’s content knowledge in: physical science, earth / space science, and life science  Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART) - National Science Foundation  Multiple-choice instrument designed to measure gains in teacher content knowledge in setting s such as summer institutes. 

10 Resources Overview of reliability and validity     Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth.  Shultz, K. S. & Whitney, D. W. (2005). Measurement theory in action: Case studies and exercises. Thousand Oaks, CA: Sage. Information on established instruments   Measuring Teacher Content Knowledge

11 Information on developing instruments  Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth.  Haladyna, T. M. (1999). Developing and validating multiple-choice test items (2 nd ed.). Mahwah, NJ: Erlbaum.  Osterlind, S. J. (1998). Constructing test items: Multiple-choice, constructed response, performance, and other formats. Boston, MA: Kluwer.  Shultz, K. S. & Whitney, D. W. (2005). Measurement theory in action: Case studies and exercises. Thousand Oaks, CA: Sage. Measuring Teacher Content Knowledge

12 When To Administer Your Test Administer the pretest immediately prior to the MSP activity or college course Administer the posttest after the MSP activity and follow-up are complete, but within the same school year as the initial MSP activity Measuring Teacher Content Knowledge

13 How to Determine Whether Teachers Have Made Significant Gains To report for GPRA, US ED needs a way to tell whether the observed gains in teacher content knowledge between the pretest and the posttest are significant US ED is proposing that projects use a statistical test called a dependent t-test to determine whether teachers have made significant gains Measuring Teacher Content Knowledge

14 Proposed process  The federal program office would supply grantees with an Excel spreadsheet with embedded formulas to do the needed calculations  Grantees would enter into the spreadsheet the pretest scores and posttest scores for the teachers they test  Grantees would use one spreadsheet per test  The spreadsheet would calculate the needed statistics and produce a report for grantees showing the total number of teachers and the number who showed significant gains  Grantees would report this information to US ED through the Annual Performance Report  US ED would aggregate the information from all grantees and use it for GPRA reporting Measuring Teacher Content Knowledge

15 Discussion Experiences developing content knowledge instruments Challenges encountered in measuring changes in content knowledge Questions about measuring changes in content knowledge