1. Promoting a Culture of Evidence and Use of Data through Program Evaluation Session Theme 2 Presentation to: OSEP Project Directors’ Conference July 21, 2009 Presentation by: Thomas Fiore, Ph.D. Project Director Center to Improve Project Performance Westat

2. 2 Session Theme 2 Evaluation tools and processes can help project directors do the best job with project planning, implementation, and accountability.

3. 3 Presentation Outline Presentation outline Keeping a project focused Logic model as the starting point Evaluation plan Evaluation design Evaluation implementation CIPP’s role providing general TA CIPP contact information

4. 4 Keeping a Project Focused Project purpose is to do a good job of the right thing. Need data to document the activities (outputs) of a project. Need data to document the overall success or lack of success (outcomes) of a project. Most important, need data to determine whether innovations should be scaled up, changed, or abandoned. Need a tool to understand which data are important—that tool is a logic model.

5. 5 Logic Model as the Starting Point A logic model... Portrays a project’s overall plan; Clarifies the relationships among a project’s goals, activities, outputs, and outcomes; and Displays the connections between those defining features of a project.

6. 6 Logic Model Structure Create a coordination hub where the various TA&D centers funded by OSEP and other federal agencies can find and share information and resources, collaborate and problem-solve together. Provide logistical support for coordination, communication and collaboration  maintaining/ expanding workgroups  developing listservs  maintaining and updating an integrated events calendar Number/types of workgroups created and maintained Number/types of listservs and listserv participants Up-to-date events calendar Goals Strategies/ Activities Outputs DirectIntermediateLong-term Outcomes An effective single point of entry for network resources is implemented/ maintained Beneficial connections exist among Network participants Successful ways of locating and sharing information and resources among network members and others are implemented /maintained The capacity of the TA&D Network members to serve clients increases steadily Inputs OSEP funding Project staff Prior experience Research-based policy and practices External Factors/Context: Other federal initiatives; OSEP policy environment; grantee’s accumulated experience and visibility.

7. 7 Logic Model Thus, logic models can be used as a starting point to plan data collection and analysis aimed at measuring project processes and performance. Systematically measuring project processes and performance is evaluation. A logic model implies a causal relationship that flows from goals to outcomes. Evaluation can be viewed as a test of the logic model’s implied hypotheses of this causal relationship.

8. 8 Evaluation Plan From the logic model, develop a plan for collecting and analyzing data. Focus on outputs and outcomes Useful for formative [define] or summative purposes [define], or both

9. 9 Evaluation Plan Develop the specific plan by answering these questions: What data are required to demonstrate project effectiveness or to provide information on the overall program effectiveness? What are the strategies/activities that should be given priority—that is, which ones should be evaluated because they are important? Who are the targeted recipients of interest and in what settings?

10. 10 Evaluation Plan Develop the specific plan by answering these questions: [more] What are the evaluation questions? What data collection activities are needed? How will the data be analyzed? What are the necessary timelines, staff assignments, and cost allocations across years? How will this be documented and reported?

11. 11 Evaluation Plan Logic model leads to evaluation questions: → Relevant goals (not necessarily all) → Salient strategies/activities related to those goals → Outputs associated with the strategies/activities → Outcomes (the most consequential ones) → Evaluation questions

12. 12 Evaluation Plan Goal: Create a coordination hub… Strategies/activities: Provide logistical support… Outputs: Number/types of workgroups, etc. Outcomes: Successful ways of locating and sharing information and resources… Evaluation question: To what extent have activities supporting coordination, communication, and collaboration been effective in enabling TA&D Network members to do their work efficiently and without duplication?

13. 13 Evaluation Design Evaluation plan leads to an evaluation design: → Evaluation questions → Measurable outputs or outcomes → Methods that capture change → Types of data collection → Instruments

14. 14 Evaluation Design Evaluation question: To what extent have activities…been effective in enabling TA&D Network members to do their work efficiently and without duplication? Measurable outcome: An integrated technology system is implemented and used. Types of data collection: Record review, survey Instruments: Record Review and Web Statistics Protocol, Annual TA&D Network Survey

15. 15 Evaluation Design Evaluation design continues with: Instrument development Sampling Data collection scheduling

16. 16 Evaluation Implementation Evaluation implementation requires: Data Collection Analysis Reporting

17. 17 Summary Sounds complicated, but projects are doing much of this already. Best when integrated into the overall implementation of the project. Best when formative and summative evaluations are integrated. Doesn’t need to be comprehensive—don’t need to measure everything in the most rigorous way to have information that can be useful. Overall, evaluation answers the question of great interest to funders and to clients: What good is this doing?

18. 18 CIPP’s Role CIPP role is two-fold: To guide, coordinate, and oversee the summative evaluations of 12 large grant-funded projects selected by OSEP. To provide technical assistance to current OSEP grantees in conducting formative and summative evaluations of their projects funded through the following programs: Parent Information Centers, Technical Assistance and Dissemination, Personnel Development, and Technology and Media Services.

19. 19 CIPP’s Role Providing General TA Vehicles for accessing TA: CIPP website (www.cippsite.org) provides an opportunity for staff of OSEP-funded projects to pose questions or ask for assistance. ◦ CIPP website contains information and reference materials on formative and summative evaluation. ◦ Evaluation Briefs will be available that discuss information requested with some frequency and that can be downloaded from the website. Toll-free telephone line (1-888-843-4101) also enables project staff to pose questions or ask for assistance.

20. 20 CIPP Contact Information Website: www.cippsite.org Toll-free telephone: 1-888-843-4101