Considering the Roles of Research and Evaluation in DR K-12 Projects December 3, 2010

Discovery Research K-12 Program Solicitation All DR K-12 proposals must have a plan for formative and summative evaluation of project research and development work. The evaluation should focus on the validation of, fidelity to, and the usefulness of the development and research processes to achieve the targeted outcomes. Individually write down: One idea you are confident you know about evaluation of research in DR K-12 projects, and One question you have about evaluation of research in DR K-12 projects

Discuss: 10 minutes At your tables, share with your group the two things you wrote down As a group, identify one idea and one question to share with the room

Discovery Research K-12 Project Evaluation Projects focused at different stages in the DRL cycle of research and development will naturally choose different development and implementation strategies and different research and project evaluation methods. The research, development, implementation and evaluation activities need to be appropriately rigorous.

Discovery Research K-12 Project Evaluation The objectives of the evaluation include: (1) ensuring that the project is making satisfactory progress toward its goals; (2) recommending reasonable, evidence-based adjustments to project plans; (3) determining the value of the outcomes of the project; and (4) attesting to the integrity of outcomes reported by the project.

DR K-12 Project Evaluation Roles, by Project Component and Evaluation Purpose Project Component Evaluation Purpose Design Critique Critical Friend Documenting Processes Formative Feedback Summative Judgment Development of Materials/PD/ Instruments Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Int Evaluator Ext Evaluator Adv Board (Quality of Process and Products) Implementation/ Impact of Materials/PD Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Int Evaluator Ext Evaluator (Extent, Nature, and Conditions of Use and Effectiveness) Research Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Adv Board Int Evaluator Ext Evaluator Int Evaluator Ext Evaluator Adv Board (Audit/Review of Research Quality)

An Example For two projects that are developing and testing mathematics teacher professional development programs, the evaluation: Works with the projects to devise logic models of the assumptions, activities, and intended outcomes of the PD Uses the logic model in Design Critique and Critical Friend roles to raise concerns and offer considerations about conceptual framing, sampling, instrumentation, data collection, analysis, and interpretation of findings

An Example For a project that is studying efficacy of a mathematics teacher professional development program in settings with large proportions of ELL students, the evaluation: Documents processes by which the project is preparing facilitators to use the PD program Documents processes in terms of the nature of implementation of the program in field sites, both in terms of fidelity to the PD program and adaptation to the particular opportunities and challenges of the local ELL contexts

An Example For a project that is studying impact of school-based mathematics coaches on teaching and learning in elementary schools, the evaluation: Audits the research by conducting standards of evidence reviews of the draft research report, and coordinating preliminary peer reviews by selected experts in the field

Evaluation of Research Components Task You will receive two handouts describing hypothetical DR K-12 projects, with an emphasis on the project’s research components Decide together to begin with one of the two examples

Evaluation of Research Components Task: 30 minutes Consider together the important evaluation purposes for the project’s research component Discuss the benefits/drawbacks of having different parties (internal, external, advisory board) serve each evaluation purpose you identify for the project Repeat for the other project

Wrap-Up What revisions or additions might you now suggest for the matrix of evaluation roles by project component and evaluation purpose? Remaining questions about evaluation of research components of DR K-12 projects.