1. SYNTHESIS OF EVALUATION CAPACITY BUILDING LITERATURE
Susan N. Labin, Ph.D.
Jennifer Duffy, M.A.
Duncan Meyers, M.A.
Abe Wandersman, Ph.D.
Cathy Lesesne, Ph.D.

2. ECB DEFINIED
Evaluation capacity building (ECB) is an intentional process which aims to increase motivation, knowledge, skill, or structural resources to conduct or use evaluation activities.
ECB can seek change at the individual, organizational, or community level.*
*Can be at national level but these efforts are not included in this synthesis.

3. FOUNDATION OF EXISTING ECB FRAMEWORKS
Evaluation Capacity
Cousins, Goh, Clark & Lee (2004)
Milstein & Cotton (2000)
Owen (2003)
Preskill & Boyle (2008)
Taylor-Ritzler, Garcia-Iriarte, & Suarez-Balcazar (2008)
Empowerment Evaluation
Chinman et al. (2008)
Fetterman & Wandersman (2005)
Wandersman (2009)

4. WHY A LOGIC MODEL FRAMEWORK?
Include and integrate concepts from multiple sources
Logic models are widely used
Include essential elements when reviewing and coding articles
Check the fit between existing concepts and empirical research

5. ECB DESCRIPTIVE MACRO LOGIC MODEL Long term/Sustainable
I. WHY - NEED
Reasons:
Who Requesting
Assumptions
Expectations
Goals-Objectives
Resources & strengths
I. WHAT - ACTIVITIES
Strategies
Implementation
Evaluation
III. RESULTS -OUTCOMES
Short &
Long term/Sustainable
Individual
Organizational
Negative outcomes
Lessons Learned
*Italicized elements were reported with such a high degree of inconsistency that valid and reliable data could not be collected. Thus, these items are not included in the synthesis nor are the related evaluation questions included.
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009 5

6. ECB DESCRIPTIVE SPECIFIED LOGIC MODEL
I. WHY - NEED
Reasons
Who: Internal-External
Assumptions
Expectations
Goals-Objectives
Resources & strengths
Resources: Material, Technological, Evaluation Expertise, Financial
Individual Attitudes
Organizational processes, policies, practices (PPP); culture; leadership; & mainstreaming
I. WHAT - ACTIVITIES
Strategies:
Level: individual-organizational
Mode; content; theory; timing; frequency; dosage
Implementation:
Target population; type of org.
Contextual: needs assessment; tailor to context; mid-course corrections
Barriers
Evaluation of ECB
Theories, design; methods; measures; timeframe; who is conducting.
III. Results-Outcomes
Short &
Long term/Sustainable
Individual
Attitudes; knowledge;
skills.
Organizational
Org PPP; culture; leadership; mainstreaming; resources
Negative outcomes
Lessons Learned 6
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009 6

7. Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009
EVALUATION QUESTIONS
What strategies, implementation variables, evaluations, and outcomes are being reported in the empirical ECB literature?
For example:
What types of individual and organizational level strategies are being used, what content is being targeted?
What types of evaluations are being reported, e.g. theories, designs, data collection methods, measures?
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

8. Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009
SYNTHESIS METHOD
Meta-analytic principle-systematic decision rules
Broad-based: included broad array of information from all designs.*
Evaluation Syntheses from Government Accountability Office (GAO)
Framework, e.g. logic model
Evaluation Questions
Guide to Community Preventive Services from CDC and used at USDHHS and other agencies
*Labin, S Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, , (eds) Nick L. Smith and Paul R. Brandon, Guilford Press, NY, NY.
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

9. STEP 1: INCLUSION CRITERIA AND SEARCH PROCEDURES
Criteria: our definition of ECB, empirical, sufficient info on ECB
Databases Searches-yielded final sample of 79 cases
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

10. Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009
STEP 2: EXTRACT AND CODE
Mostly closed-ended coding form
Coding manual for reliability among 3 coders
Tested reliability sample between coders:
Kappa =.83; average Kappa =.66
Inter-coder reliability =70-95%; average 85%
Reliable depiction of literature
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

11. STEP 3: ANALYSIS
Efforts and interested audiences likely differ for Broad-Based ECB efforts versus Classroom on key strategy, implementation, and outcomes
Confirmed that Broad versus Classroom empirically differed on a number of hypothesized characteristics
Decided to run analyses separately for the two groups

12. STEP 3: ANALYSIS BROAD VERSUS CLASSROOM
Target Pop-level
Ind & Org
Individual
Target Pop-type
Staff
Students
Types
Training/TA
Classroom
Org Strategies
Yes No
Outcome
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

13. STEP 3: ANALYSIS HIGHLIGHTS OF FINDINGS
Results for two groups were mostly similar
Almost a third exclusively in Health, one quarter exclusively in Education, and some in multiple domains (20% Broad; 11% Classroom).
About half had some evaluation of ECB
Eval Data Collection Methods: nearly half surveys and over 10% focus groups; but more Broad used interviews (31% v 16%) and other types , e.g. 3rd party observations & document review
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

14. EVALUATION METHODS FINDINGS cont.
Evaluation Design
Over half “case studies”- multiple & in-depth methods
No Experimental (random)
Three quasi-experimental
Broad stronger designs
More Pre-post (15%) for Broad; Classroom (Zero)
Fewer post-only Broad (13%) than Classroom (21%)
Three cited instruments-rare discussion of measurement
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

15. STRATEGIES: UNDERLYING THEORY
BROAD
CLASSROOM
Reported Underlying Theory
80%
63%
Specified Theory
Half
Two-thirds
Of Specified : Participatory, Collaborative, &/or Empowerment (EE*)
One -third
Broad: 1 utilization; Classroom, 2 utilization theory.
*Half of these for both groups specified Empowerment Evaluation
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

16. INDIVIDUAL LEVEL STRATEGIES: CONTENT
BROAD
CLASSROOM
Seven Items: Mean
3.48
2.84
Attitudes
23%
11%
Design or Plan Eval
82%
53%
Do an Eval
69%
68%
Logic Models
21%
Concepts, approaches, methods
38%
58%
Interpret and use data
26%
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

17. ORGANIZATIONAL LEVEL STRATEGIES: CONTENT FOR BROAD-BASED EFFORTS
Reporting Content: 44%
Most frequent: Organizational processes, policy and practices: less than one third
Some: Mainstreaming, Organizational Culture, Resources
Least: Leadership: 10%
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

18. INDIVIDUAL LEVEL OUTCOMES
Attitudes-1 item:
Broad: 36%; Classroom 26%
Knowledge-6 items:
Broad: 51%; Classroom 53%
Concepts, Approaches, Methods
Hire/work w evaluator (3)
Logic Models
Plan or Design Eval
How to Do An Evaluation
Interpret and Use Data
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

19. ORGANIZATIONAL LEVEL OUTCOMES BROAD-BASED EFFORTS
Item
% Reporting
# Items
Mean
Org Outcomes Overall
70% 17 6
Process, Policies, Practices (PPP)
72% 7
2.8
Resources
48% 5
1.9
Org Culture
31% 2
1.6
Leadership
13% 1
Mainstreaming
53%
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

20. LESSONS LEARNED (LL) WHAT’S IMPORTANT?
Broad-Based
Interpersonal skills, relationship building, org culture, leadership, funder support and attitudes.
Classroom
Interpersonal relationship building , need for ongoing training or TA
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

21. PRELIMINARY CONCLUSIONS
Need more targeting of attitudes? Minimal targeting of attitudes; attitudes reported as barrier; positive attitude outcome could be stronger (1/3); LL interpersonal relations.
Need more targeting of leadership and org culture? Minimal targeting of leadership & org culture; leadership only sometimes a strength (1/3 Broad); LL importance of culture & leadership.
Not strong measurement or design in evaluating ECB efforts.
Frequent reporting of ind & org level outcomes for Broad ECB efforts
Frequent reporting of ind level outcomes for classroom ECB efforts
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

22. Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009
SO WHAT ? WHAT NEXT?
Develop and use consistent description and measurement of ECB efforts and results - start with categories and subcategories used here for scale development
Consistent description and measurement would be basis for more consistent reporting and synthesis and use of findings
Pay more attention to evaluation design, e.g. pre-post and natural comparison groups
Consider theory and strategies for enhancing leadership and organizational culture
Explore possible sequential pattern to achieving organizational PPP (processes, policies, practices), mainstreaming, leadership, and culture changes.
Include funders in ECB training and learning efforts
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009

23. THANK YOU For questions or comments please contact Susan Labin, Ph.D.
Labin,Dufy,Meyers,Wandersman, Lesesne-AEA-Nov.14,2009