t 1 Developing Measures of Mathematical Knowledge for Teaching Geoffrey Phelps, Heather Hill, Deborah Loewenberg Ball, Hyman Bass Learning Mathematics for Teaching Study of Instructional Improvement Consortium for Policy Research in Education University of Michigan MSP Regional Conference Boston, MA March 30-31, 2006

t 2 Overview of today’s session 1. LMT/SII Measures Development 2. Some Sample Results 3. LMT/SII Measures and Dissemination

t 3 Subtract: What is “Content Knowledge for Teaching”? An Example From Subtraction

t 4 Analyzing Student Errors

t 5 Analyzing Unusual Student Solutions

LMT/SII Measures Development

t 7 Why Would We Want to “Measure” Teachers’ Content Knowledge for Teaching? To understand “what” constitutes mathematical knowledge for teaching To understand the role of teachers’ content knowledge in students’ performance To study and compare outcomes of professional development and teacher education To inform design of teachers’ opportunities to learn content knowledge

t 8 Measuring Teachers’ Mathematics Knowledge: Background and History Research on teacher behavior Early research on student achievement –Proxy measures for teacher knowledge –Tests of basic skills 1985 on: “the missing paradigm” pedagogical content knowledge or PCK 1990s: interview studies of teachers’ mathematics knowledge (MSU -- NCRTE)

t 9 Study of Instructional Improvement Study of three Comprehensive School Reforms; teacher knowledge a key variable Instrument development goals: –Develop measures of content knowledge teachers use in teaching K-6 content for elementary school teachers Not just what they teach - specialized knowledge –Develop measures that discriminate among teachers (not criterion referenced) –Non-ideological But we faced significant problems….

t 10 Problems As We Began This Work No way to measure teachers’ content knowledge for teaching on a large scale –Small number of items, many written by Ball, Post, others appeared on every instrument –Nothing known about the statistical qualities of those items (difficulty, reliability) –Studies relied on single items, yet single items unlikely valid or reliable measures of teacher knowledge

t 11 Early Decisions and Activity Survey-based measure of CKT-M –3000 teachers participating in SII –Multiple choice Specified domain map 5 people + 5 lbs cheese + 5 weeks = 150 items (May 2001) Large-scale piloting, summer 2001

t 12 Early Decisions and Activity Types of knowledge Mathematical content Content knowledgeKnowledge of content and students Number Operations Patterns, functions, and algebra

t 13 Early Analyses and Validity Checks Results from piloting –We can measure teachers’ CKT-M –Reliabilities of –Factor analysis shows distinct types of knowledge Knowledge of content and students (KCS) separate from CK Specialized content knowledge (SCK) vs. common content knowledge (CCK) Hill, H.C., Schilling, S.G., & Ball, D.L. (2004) Developing measures of teachers’ mathematics knowledge for teaching. Elementary School Journal 105,

t 14 Reliabilities (1PL-IRT): Elementary Knowledge of contentKnowledge of content and students Number and operations (K-6) Patterns, functions, and algebra (K-6) Geometry (3-8)

t 15 Reliabilities (1PL-IRT): Middle School Knowledge of contentKnowledge of content and students Number and operations (5-9) Patterns, functions, and algebra (5-9) Geometry (3-8)

t 16 Content Knowledge : Number and Operations Common knowledge –Number halfway between 1.1 and 1.11 Specialized knowledge –Representing mathematical ideas and operations –Providing explanations for mathematical ideas and procedures –Appraising unusual student methods, claims, or solutions

t 17 Representing Number Concepts Mrs. Johnson thinks it is important to vary the whole when she teaches fractions. For example, she might use five dollars to be the whole, or ten students, or a single rectangle. On one particular day, she uses as the whole a picture of two pizzas. What fraction of the two pizzas is she illustrating below? (Mark ONE answer.) a) 5/4 b) 5/3 c) 5/8 d) 1/4

t 18 Providing Mathematical Explanations: Divisibility Rules Ms. Harris was working with her class on divisibility rules. She told her class that a number is divisible by 4 if and only if the last two digits of the number are divisible by 4. One of her students asked her why the rule for 4 worked. She asked the other students if they could come up with a reason, and several possible reasons were proposed. Which of the following statements comes closest to explaining the reason for the divisibility rule for 4? (Mark ONE answer.) a) Four is an even number, and odd numbers are not divisible by even numbers. b) The number 100 is divisible by 4 (and also 1000, 10,000, etc.). c) Every other even number is divisible by 4, for example, 24 and 28 but not 26. d) It only works when the sum of the last two digits is an even number.

t 19 Which of these students is using a method that could be used to multiply any two whole numbers? Appraising Unusual Student Solutions

t 20 Common vs. Specialized CK Appears in exploratory factor analyses on 2/7 forms; confirmatory on 3/7 Individuals can be strong in common but not specialized; vice versa Support from cognitive interviews of mathematicians Suggests there is professional knowledge for teaching

t 21 Ongoing Work Item and measures development –Middle school national probability study –Develop new measurement modules for data analysis and for probability Validation efforts –“Videotape” study –Cognitive tracing studies –Content validity checks

Some Sample Results

t 23 An Example : Establishing a Relationship to Student Growth

t 24 Links to Study of Instructional Improvement Student Achievement Analysis SII CKT-M measure – 38 items –SII:.89 IRT reliability Model: Student Terra Nova gains predicted by: –Student descriptors (family SES, absence rate) –Teacher characteristics (math methods/content, content knowledge) Teacher content knowledge significant –Small effect (LT 1/10 standard deviation) –But student SES is also on same order of magnitude Hill, H.C., Rowan, B., & Ball, D.L. (2005) Effects of teachers' mathematical knowledge for teaching on student achievement. American Educational Research Journal 42,

t 25 A Second Example : Evaluating Teacher Professional Development

t 26 Tracking Teacher Growth Items piloted in California’s Mathematics Professional Development Institutes (MPDI) –Instructors: Mathematicians and mathematics educators – hours of professional development –Focus is squarely on mathematics content –Summer 2001 –Pre/post assessment format (parallel forms) Hill, H. C. & Ball, D. L. (2004) Learning mathematics for teaching: Results from California’s Mathematics Professional Development Institutes. Journal of Research in Mathematics Education 35,

t 27 MPDI Teacher Growth (Year 1) For all institutes for which we have data, teachers gained.48 logits, or roughly ½ standard deviation Translates to 2-3 item increase on assessment Considered substantial gain

t 28 Results from Sample Institutes

t 29 MPDI Evaluation: Other Findings Length of institute predicts teacher gains –120-hour institutes most effective, on average –But some 40-hour institutes very effective Focus on mathematical analysis, proof, and communication leads to higher gains Many questions remain –Effects of content (e.g., mathematics vs. student thinking) –Treatment of content: common vs. specialized –Effects of teacher motivation –Long term learning from colleagues, curriculum, practice

LMT/SII Measures and Dissemination

t 31 Current Item Pool Equated forms for elementary school: –Number & operations / Content knowledge (K-6) –Number & operations/ Knowledge of content and students (K-6) –Patterns Functions & Algebra/ Content knowledge (K-6) –Geometry (3-8)

t 32 Current Item Pool Equated forms for middle school: –Number & operations / Content knowledge (5-9) –Patterns Functions & Algebra/ Content Knowledge (5-9) –Geometry (3-8)

t 33 Item Workshops and Dissemination Interested users attend a one-day workshop in Ann Arbor We cover –History of item development –Analytic methods and validation studies –How to use technical materials Users get –Access to measures –Support materials

t 34 Dates and Contact Information Learning Mathematics for Teaching – Dates for LMT Workshops –May 19, 2006 –August 10, 2006 –Brenda Ely Geoffrey Phelps –