Enhancing Evaluation Stakeholder Responsiveness Through Collaborative Development of Data Collection Instruments Karen Kortecamp, PhD The George Washington University

Collaborating With Stakeholders Should we or shouldn’t we?  Utilization-focused evaluators  Accommodate stakeholder concerns  Increase evaluation utilization  Theory-focused evaluators  Minimize the effects of stakeholder influence  Apply preset standards  Social justice perspective  Evaluation should be responsive to the views of the weakest stakeholders

Factors to Consider  Type of program being evaluated  Size of program  Duration of program  Number of stakeholders involved  Impetus for the evaluation  Internal or external evaluation  Implications of the evaluation  Point at which the evaluation of a program is designed and implemented  Funding available to conduct the evaluation

Evaluation Context  Initial grants funded in 2001  Teacher professional development  Ultimate aim is increased student achievement in American history Grant that is the subject of this presentation: Large school system Voluntary participation of 8 th & 9 th grade teachers Three cohort groups serving up to 75 teachers U. S. Department of Education required: External evaluation Measure of teachers’ knowledge Quasi-experiment

Project Objectives 1) Increase teachers’ knowledge of traditional American history 2) Increase teachers’ use of primary sources in teaching traditional American history 3) Increase student knowledge of and interest in traditional American history

The Collaborative Process Decision-making stakeholders ◦ Project Manager, project coordinator, lead historians and faculty of partner university Theory of change ◦ Amalgamation of assumptions, experiences, and intuition Evaluator role in articulating a theory? ◦ Asking good questions

Pricking the Balloon! What specific needs will the project address? How? Why the interventions identified? What specific change(s) are desired? How can change(s) be measured? Are there barriers to overcome? What do you most want to know? What outcomes do you most want to occur?

Collaborating in Developing Evaluation Instruments Development of 5 Primary Instruments ◦ Assessment of teachers’ knowledge ◦ Survey of teachers’ perceptions ◦ Observation of teachers’ practice ◦ Assessment of students’ knowledge ◦ Survey of students’ attitudes Additional data collected through focus group interviews, event surveys, and analysis of teacher reflections & products

Guiding Development of the Teacher Assessments What knowledge is of most value? What knowledge will the professional development address? What constitutes evidence that knowledge has been acquired and applied? Stakeholders developed content Evaluator focused on developing valid and reliable measures

An Early Draft of the 8 th Grade Teacher Assessment Included multiple choice questions that had nothing to do with the professional development content Asked teachers to: “Briefly explain why middle school students should study American history” Asked teachers to observe a Civil War period political cartoon and develop 2 or 3 questions they would ask their students about this primary source

Some of the Problems Gathering data on content not related to the professional development would yield meaningless data Scoring teachers’ responses as to why middle school students should learn American history would be highly subjective (impossible) and not relevant The direction to develop 2 or 3 questions was vague; scoring would be a nightmare What would any of these items tell stakeholders about the influence of the professional development on teachers’ knowledge? What would the assessment tell the U.S.DOE about the effectiveness of the project?

Final Iteration Represented content addressed in the professional development Integrated assessment of teachers’ historical thinking Measured teachers’ ability to apply historical thinking to guide students’ analysis and interpretation of primary sources

Assessment Items Content pairs ◦ Benjamin Franklin – Slavery Primary and secondary sources ◦ Define and provide examples of both; identify steps historians take to analyze and interpret primary sources Analysis of a primary source ◦ Political cartoon from a historical period Develop student questions ◦ Guide students in analyzing and interpreting meaning

Conclusions The collaborative process: ◦ supported stakeholders in developing a shared vision ◦ sharpened stakeholders’ focus on outcomes ◦ deepened stakeholders understanding of the central purpose of evaluation ◦ led stakeholders to utilize evaluation findings to strengthen the professional development programs ◦ enhanced the evaluator’s understanding of the assumptions and beliefs that guided decision- making ◦ promoted mutual trust and respect