TAH Project Evaluation Great - Yet Realistic - Expectations Jeff Sun, Sun Associates

Our Background Initiative-wide Qualitative Evaluation District-level initiatives State-level initiatives Strategic Planning STEM, Science, Instructional Technology, Social Studies, New Media Evaluation of Teacher Professional Development 10 TAH projects (7 of these are current)

Qualitative is the Emphasis Makes the connection between what projects say they want to do and what participants say that projects have done. This data is invariably qualitative and accounts for a wide range of effects Quantitative data helps paint the picture in broad outlines......but qualitative data helps you understand what the quantitative data means and to use the evaluation strategically

This Presentation We want to talk about how to Create a good evaluation Use evaluation strategically Work effectively with evaluators

Four Functions for Evaluation Evaluation performs four meaningful functions in a project’s life... As a part of the proposal To clarifying a project’s work To provide formative assessment of Progress and Implementation Summatively, to account for the project’s success (or lack thereof)

First Function - The Proposal As expressed in the logic map, evaluation helps explain: Need for the project The project’s actions in response to those needs Who the project will serve What the project expects to produce All in one tidy package in the proposal text!

Issues Related to Function 1 We need time to develop the evaluation plan Ideally, we come in when the basic work is set, but the details are not fully fixed We need space to fully describe the evaluation process It’s hard to create this plan in 2 paragraphs or a half page We need connect with those who will be actually implementing the project if funded Proposal-writers will help you get funded...but it’s often no coincidence that they’re nowhere to be found once you get the grant.

Second Function - Clarification Once you’re funded, the real work begins Use the evaluation plan as an organizer for talking with partners about what the project will be doing expectations for partners and participants This discussion clarifies goals and develops a shared vision for project success Out of this discussion come the project’s performance indicators...which we organize into rubrics It would be nice to have all of your indicators before funding, as part of the proposal But realistically, there’s never time for that Projects need their own indicators ED’s indicators are not the same and cannot replace your own indicators

Issues Related to Function 2 We can’t help if we aren’t at the table Often, we have to really bug projects to bring us on board early Evaluation is too often seen as the thing that happens after the work is done...not as the thing that can get work done. We read the proposal and base the actual evaluation on what is written This can be a problem when the proposal was written by the now-absent proposal-writer The clarification discussion - assuming it happens early enough - is where this problem is resolved

Third Function - Formative Collecting the data and sharing it as the project progresses Measuring what the project is doing against what it said that it would do Using indicator rubrics to account for data that is highly qualitative

Issues Related to Function 3 Our challenge is to identify data collection points that get at the essence of the project and its impact. This can be one of the most important parts of our work. Questions continue the process of clarifying the project work through participant feedback on impact Next most difficult is connecting directly with participants

Fourth Function - Summative Annual comparison of progress to goals Sometimes more reporting than “annual” is important More reporting might be useful in the beginning, less reporting might be necessary as the project matures Summative data should play a role in the development of new proposals

Issues Related to Function 4 Qualitative data is valuable for projects, but does not seem to be something that ED is interested in. This is a problem because projects cannot really produce meaningful data that ED is interested in. So, do we spend extra time (money) producing two sets of data? Or do we sacrifice the meaningful data to concentrate on producing what we’re told to create? This has been the on-going debate among TAH evaluators on a national level

Questions? Comments? What challenges have you faced when considering project evaluation? How have you used evaluation in different stages of your project? (from planning to reporting) or x 204