American Evaluation Association Developing A Method For Classifying Evaluation Activities in a Foundation Thursday, November 7, 2002 American Evaluation Association Washington, D.C.

Overview 1. Context 2. Key Questions 3. A Methodology 4. Use in Managing Evaluation Activities 5. Summary 6. Future Questions

1. Context ...at the Foundation From January 2000 to December 2001 about 1,800 grants about $ 900,000,000 granted Six major program areas In 2001, evaluation department established

2. Key Questions What does the portfolio of evaluation activities look like? How can we classify the various evaluation activities at the Foundation? What categories are useful for deciding how to manage evaluation activities?

3. Developing A Method Identified Key Questions Developed Classification Definitions 1. Program Area 2. Budget 3. Evaluation Design: causal or descriptive 4. Report Author: third party or program operator 5. Evaluation Funder: foundation, program operator, or other 6. Deliverable Due Date

Additional Definitions Grant Grouping: “line of work” or individual Pyramid Level: 1 through 6 Evaluation Purpose: summative, formative, research, monitoring. Primary Audience Primary Program Funder Strategy Year Program Started 3. Method (continued)

Collected Data Grants Database (GIFTS) Grants Summaries Grant Proposals Board Docket Consultant Contracts Program Staff 3. Method (continued)

Reviewed Findings Iteratively with Program Staff Analyzed Data Reviewed Findings Iteratively with Program Staff Definition refinement Re-collection of data Re-analysis of data 3. Method (continued)

4. Use in Managing Evaluations To focus evaluation department efforts. To guide decisions about what to evaluate. To develop a reporting schedule to plan discussions and learning opportunities about program performance and improvement. To improve the design of the grants database (GIFTS).

5. Summary ”Dimensions” of the portfolio of evaluation activities at the Foundation. Evaluation activities were classified along more than ten dimensions: budget, evaluation design, author, etc. Categories most useful for managing include program area, budget, design, deliverable due date.

6. Future Questions What other criteria should be considered when deciding how to classify evaluation activities? What other criteria should be considered when deciding how to manage an evaluation portfolio? (grant characteristics and/or grant contexts) How have other grant-making foundations classified their evaluation activities?