Evaluating Professional Development Debbie Junk, Coordinator for Mathematics Initiatives Mathematics Project Directors’ Meeting Tuesday, October 9th Austin, Texas Debbie Junk, Coordinator for Mathematics Initiatives Mathematics Project Directors’ Meeting Tuesday, October 9th Austin, Texas

 True professional development should be a learning experience for all who are involved…professional development is a purposeful and intentional process designed to enhance the knowledge and skills of educators so that they might, in turn,improve the learning of all students.” (Guskey, 2001 p. 121) What counts as professional development?

Folding Activity

Why do we evaluate?  Effectiveness of professional development in mathematics need to be evaluated to assist in-  Planning  Formative assessment and,  Summative assessment  To provide evidence of effectiveness so that good programs get better, great programs spread and ineffective programs get eliminated or significantly reworked.  Effectiveness of professional development in mathematics need to be evaluated to assist in-  Planning  Formative assessment and,  Summative assessment  To provide evidence of effectiveness so that good programs get better, great programs spread and ineffective programs get eliminated or significantly reworked.

Whose Fault is it?  Evidence versus proof  The complex nature of teaching and learning  Evidence versus proof  The complex nature of teaching and learning

Effectiveness indicators show:  Increased student achievement  Increased teacher content knowledge  Evidence that shows the two are linked in some way.  Increased student achievement  Increased teacher content knowledge  Evidence that shows the two are linked in some way.

What Counts as Mathematics Knowledge for Teaching (MKT)?  Mathematics teachers need to know the math they teach in a very specialized way-  Children’s thinking  Task posing  Assessment  Expectations  Decision-making  Mathematics teachers need to know the math they teach in a very specialized way-  Children’s thinking  Task posing  Assessment  Expectations  Decision-making

Research  Teachers who score high on knowledge assessments designed to test this specialized knowledge have students who learn more in mathematics in their classrooms. Ball, et al, 2004  Students who are in classrooms with teachers who understand and can predict their thinking in mathematics score higher on achievement tests. Carpenter, et al 1998  Teachers who score high on knowledge assessments designed to test this specialized knowledge have students who learn more in mathematics in their classrooms. Ball, et al, 2004  Students who are in classrooms with teachers who understand and can predict their thinking in mathematics score higher on achievement tests. Carpenter, et al 1998

What kinds of evaluation should be given?  Levels of evaluation  Participants’ reactions  Participants’ learning  Organization support and change  Participants’ use of new knowledge and skills  Student learning outcomes From Guskey, 2001 pp  Levels of evaluation  Participants’ reactions  Participants’ learning  Organization support and change  Participants’ use of new knowledge and skills  Student learning outcomes From Guskey, 2001 pp 79-81

You get what you ask for:  After a 2-day Workshop on Cooperative Learning, one participant responded, “The ideas were fine, but they had us working in groups too much”.  One teacher responded to a workshop titled, Tactics for Thinking: “this is all very interesting but I feel it requires students to think too much!”  From Gusky, 2001  After a 2-day Workshop on Cooperative Learning, one participant responded, “The ideas were fine, but they had us working in groups too much”.  One teacher responded to a workshop titled, Tactics for Thinking: “this is all very interesting but I feel it requires students to think too much!”  From Gusky, 2001

Designing and/or choosing the instrument  Understand the purpose of your assessment-  Understand the intervention you are assessing-  The questions should vary in scope so that you gain understanding in all areas of Math Knowledge for Teaching.  Understand the purpose of your assessment-  Understand the intervention you are assessing-  The questions should vary in scope so that you gain understanding in all areas of Math Knowledge for Teaching.

How much is enough?  Practical considerations  Length of the evaluation tool  Time, when and where  Pre and post data  Participant numbers  Workshop numbers  Practical considerations  Length of the evaluation tool  Time, when and where  Pre and post data  Participant numbers  Workshop numbers

What do you do with ALL THAT DATA?  Ethical considerations  Permission to use/collect data  Anonymous vs. named  Paper assessments  Electronic assessments  Summarizing and analyzing assessment data  Ethical considerations  Permission to use/collect data  Anonymous vs. named  Paper assessments  Electronic assessments  Summarizing and analyzing assessment data

Qualitative or Quantitative… Qualitative evaluations of program effectiveness can occur at any level in many formats.  interviews  open-ended responses  observations  journals Summarizing these data usually takes more time and consistently reporting the results can be difficult. Information gained from qualitative data can yield a more personal and deeper understanding of the participants’ experiences. Qualitative evaluations of program effectiveness can occur at any level in many formats.  interviews  open-ended responses  observations  journals Summarizing these data usually takes more time and consistently reporting the results can be difficult. Information gained from qualitative data can yield a more personal and deeper understanding of the participants’ experiences.

Quantitative Data  Easy to collect  Easier to interpret without bias*  Limited in scope  Likert scales  Multiple choice *bias is still present, especially within the assessment design  Easy to collect  Easier to interpret without bias*  Limited in scope  Likert scales  Multiple choice *bias is still present, especially within the assessment design

Both?  Typical professional development evaluation has both elements.  e.g. “Were your expectations met today?” (a numeric score is assigned)  Tell what you liked best about this workshop--(open-ended)  Typical professional development evaluation has both elements.  e.g. “Were your expectations met today?” (a numeric score is assigned)  Tell what you liked best about this workshop--(open-ended)

A Whole Different Animal!  Assessment of learning requires a documentation of change over time.  For students, we can compare year to year test data.  Assessing teachers knowledge is more complex.  Assessment of learning requires a documentation of change over time.  For students, we can compare year to year test data.  Assessing teachers knowledge is more complex.

Mathematics and Science Partnership Grants (MSP)  Expectations

Texas Regional Collaboratives  Plan of Action:  provide instruments for use in pre- post format (e.g. Geometry, K-8 geosciences)  form advisory committee to explore evaluation and assessment  Plan of Action:  provide instruments for use in pre- post format (e.g. Geometry, K-8 geosciences)  form advisory committee to explore evaluation and assessment

HOMEWORK Evaluating Professional Development By Thomas Guskey Corwin Press, 2001 Evaluating Professional Development By Thomas Guskey Corwin Press, 2001