project evaluations a framework for your own project

who needs a project evaluation anyway? If your thesis/dissertation involves the implementation of a project or device and you want to prove its effectiveness If you are conducting an experiment these might be helpful guidelines If you are conducting a workshop or a teaching seminar Federally funded projects must be evaluated (Government Performance and Results Act, GPRA, 1993) –If you have an NSF grant to do a project

why do project evaluations? Federally funded projects must be evaluated (Government Performance and Results Act, GPRA, 1993) –If you have an NSF grant to do a project proves causal connections provides information on how to improve a project allows you to understand if goals were met can account for unanticipated consequences can better communicate to stakeholders

types of evaluations see also visual on board 1)Formative evaluation: begins during project and continues throughout project life a)Implementation evaluation: is the project going as planned? b)Progress evaluation: is the program meeting its goals? (most important in the beginning stages of the project) 2)Summative evaluation: did the project meet its stated goals? Similar to progress evaluation, but at the end of project “When the cook tastes the soup, that’s formative. When the guests taste the soup, that’s summative.”

the big picture 1)Develop a conceptual model of the project* 2)Identify project goals* 3)Define evaluation questions and measurable outcomes* 4)Develop evaluation design* 5)Collect data 6)Analyze data 7)Provide information to interested audiences *the rest of the presentation only focuses on 1-4

1)Develop a conceptual model and 2) identify project goals The logic model Inputs Activities Short-Term Long-Term Fill in the model Set up a timeframe for project Sometimes helpful to work backwards (start with long-term outcomes)

1) my conceptual model and project goals Inputs: time to plan, help from other TAs and Donna, research guidance from Ken Activity: in recitation sections, TAs choose one of three activities and do the activity with the class to discuss either a) ecotourism, b) participatory development, or c) the causes of poverty Short-term outcomes: students actively participate in the activities, explore issue a, b, or c and discuss/reflect on the issues with other students Long-term outcomes: students are more aware of international development issues and will continue to learn about them outside of class

3) Define measurable outcomes and evaluation questions four steps 1)Identify key stakeholders ( see pg 7 the NSF Guide ) 2) Formulate evaluation questions that stakeholders might ask -may produce more than you’ll actually use, but that’s OK 3) Define outcomes in measurable terms -use the questions generated in step two to fill out number three in the Goals & Objectives worksheet ( see pg 9 ) 4) Prioritize & eliminate questions -consider your most important stakeholders first -what is success? Statistical significance, etc -consider resources: time, money, staff capability (general rule: 5-10% of cost for evalutions

3) measurable outcomes and evaluation questions Example of stakeholder questions Measurable outcomes: Example of goals worksheet Prioritize: example of survey

4) Develop an evaluation design Select methodological approach and data collection instruments –Data collection matrix ( see pg 15 ) Determine who will be studied and when –randomized samples are best –Comparison groups if possible –If not, maybe levels of implementation?

evaluation design student surveys ( ) TA surveys (8) professor survey (1) any possible follow-up interviews with any of the above three must acknowledge the parameters of my samples

references NSF 2002: The 2002 User-Friendly Handbook for Project Evaluation NSF 1997: The User-Friendly Handbook for Mixed Method Evaluations Stake, R. 1957: Program Evaluation, Particularly Responsive Evaluation Yin, R.K. 1989: Case Study Research: Design and Method