1. http://www.soe.umich.edu/lmt/ Measuring Effectiveness in Mathematics Education for Teachers Heather Hill University of Michigan School of Education Learning Mathematics for Teaching 2007 MSRI June 1, 2007

2. http://sitemaker.umich.edu/lmt/ Avoiding Arbitrariness! 16 is my favorite number

3. http://sitemaker.umich.edu/lmt/ Avoid Arbitrariness!

4. http://sitemaker.umich.edu/lmt/ Challenge Knowing you’ve added (relevant) knowledge to prospective or in-service teachers –Not going to discuss student achievement as outcome Issues to consider as you pursue understanding impact: –Getting clear on your question –Research design –Instrument selection –Comparability to other projects

5. http://sitemaker.umich.edu/lmt/ Getting clear on your question Do you want to know the effect of: –A set of materials? –A course? –Course & instructor? –Sequence of courses/instructors? Different questions imply different designs –Simplest design: What is effect of course and instructor?

6. http://sitemaker.umich.edu/lmt/ Getting clear on your question Do you want to know the effect of: –A set of materials? –A course? –Course & instructor? –Sequence of courses/instructors? Different questions imply different designs –Simplest design: What is effect of course and instructor?

7. http://sitemaker.umich.edu/lmt/ Research Design Question: What would these people have known and been able to do in the absence of our program? –Estimate difference between actual and “counterfactual” Problem: Cannot estimate with program and without program at the same time –e.g., Marcia in December WITH and WITHOUT TE401 –Random assignment provides best estimate of counterfactual –Quasi-experimental designs more possible

8. http://sitemaker.umich.edu/lmt/ Stop. Design. 1 minute: Think about how you would evaluate your work with teachers –What is your question? –How can you gather evidence about your question? 3 minutes: Share & critique with neighbors

9. http://sitemaker.umich.edu/lmt/ Best Solution: Random Assignment Problem –Rules out easiest research question: you + your materials –Treatment/random assignment of students occurs in classes –Statistical tests must be performed at the level of treatment (e.g., compare this class to that)‏ Using students = cheating by boosting your power –Need large N of classrooms or programs for statistical power Even mathematicians aren’t this prolific Another: Technically complex

10. http://sitemaker.umich.edu/lmt/ Quasi-Experimental Designs Definition: No randomization to treatment Problems: –Not causal -- always threat to inferences Selection, pre-test controls, “natural” learning –“Assignment” is still class level for some questions –But easier to implement

11. http://sitemaker.umich.edu/lmt/ Quasi-Experimental Designs Worst: –Threats: selection, no comparison, no pre-test control Second-Worst –Threats: Selection into T and C, no pre-test control T post C post

12. http://sitemaker.umich.edu/lmt/ Quasi-Experimental Designs Slightly less bad, but still not good: –Threats: “Natural” learning over time; learning from instrument; selection Good: –Threats: Selection T pre T post C post T pre C pre

13. http://sitemaker.umich.edu/lmt/ Quasi-Experimental Designs Best: –Threats: Selection –Advantage: Allows for growth modeling T3T3 C3C3 T2T2 T1T1 C2C2 C1C1

14. http://sitemaker.umich.edu/lmt/ Quasi-Experimental Design: Unit of Analysis Problem Does Not Go Away To understand YOUR effect with YOUR materials, unit of analysis can be student –E.g., comparing 32 pre/post tests To separate materials effect from instructor effect, need multiple classrooms

15. http://sitemaker.umich.edu/lmt/ Example: Quasi-Experimental Design Hill/Ball study of MPDI (2002-2003 data): –Pre/post for “treatment” group (1000 teachers in about 25 sites)‏ –Pre/post for “comparison” group (300 teachers who signed up for MPDIs but did not attend)‏ Can compare change in treatment to change in comparison –MKT instrument Compare among 25 programs

16. http://sitemaker.umich.edu/lmt/ Instrumentation Criteria: –Aligned to your program’s content –Technically checked and validated –Linked to student achievement Types of instruments: –Teacher knowledge –Teacher “practice” –Mathematical quality of teaching

17. http://sitemaker.umich.edu/lmt/ Teacher Knowledge: Multiple Choice LMT: K-5, 6-8 measures in number/operations, algebra, geometry (soon: rational number, proportional reasoning)‏ www.sitemaker.umich.edu/lmt KAT: Algebra www.msu.edu/~kat/ DTAMS: K-5, 6-8 measures in Whole Number Computation, Rational Number Computation, Geometry/Measurement, Probability/Stats/Algebra http://louisville.edu/edu/crmstd/diag_math_assess_elem_te achers.html

18. http://sitemaker.umich.edu/lmt/ Knowledge: Other Methods Kersting (LessonLab): Teacher analysis of video segments Discourse analysis, clinical interviews (e.g., TELT -- see Ball’s personal website), videos of clinical teaching experiences Home-grown tests

19. http://sitemaker.umich.edu/lmt/ Possible Instruments: Observational Of “practice”: –Reformed Teaching Observation Protocol –Horizon’s Inside the Classroom Of “mathematical quality” of instruction –LMT Mathematical Quality of Instruction –TIMSS instruments

20. http://sitemaker.umich.edu/lmt/ Plea from Meta-Analysts: Comparability Use common measures across teacher education efforts. Why? –Knowledge is built by comparing effects of different programs Knowing that program A has a.5 effect is good But knowing that Program A =.5 and Program B =.3 is better; can ask what aspects of program A “worked” Must do with large “N” of programs

21. http://sitemaker.umich.edu/lmt/ Comparison Example Example: Carnegie (Matt Ellinger)‏ –Formative assessment (feedback to programs involved)‏ –Four programs with math/math ed collaboration Seven sections –Place value is content focus –LMT instrument focused on place value is pre/post –No comparison/control; internal variation

22. http://sitemaker.umich.edu/lmt/ Comparison Example Mathematical Education of Elementary Teachers (Raven McCrory)‏ –37 sections, 27 instructors, 13 institutions –588 total matched-pair student responses –Can compare outcomes by program characteristics Instructor surveys of topics taught Textbook used, chapters covered Cognitive demand measure (based on Adding It Up)‏ Instructor characteristics

23. http://sitemaker.umich.edu/lmt/ Randomized Example: Hill (fall 2007)‏ Video pre Lesson Study Math Content Coaching Records of Practice Video post

24. http://sitemaker.umich.edu/lmt/ Conclusion Don’t be arbitrary Link to many instruments described here –www.sitemaker.umich.edu/lmtwww.sitemaker.umich.edu/lmt Good design advice: –Institute for Social Research: Robin Jacob (rjacob@umich.edu)‏rjacob@umich.edu – Local university-based evaluators