1. Elements of evaluation quality: questions, answers, and resources
Leslie Goodyear, PhD
Principal Evaluation Director, EDC
NIH SciEd 2019: Annual Conference for NIH Science Education Projects April 22, 2019

2. Intro and mention of expertise in the room.
Quick hands-up activity – who’s an evaluator? Who’s not?

3. questions
 NEIL
What is it, who should be concerned about it, and what do I need to know?

4. What is evaluation?
First, let’s talk about evaluation – what is it? The systematic investigation of the quality of programs, projects…and/or their components, together or singly For purposes of decision making, judgments, conclusions, findings, new knowledge, organizational development, and capacity building in response to the identified needs of identified stakeholders Leading to improvement and/or accountability… Ultimately contributing to organizational or social value.

5. What is evaluation quality?
Who determines it? What constitutes it? Who contributes to it? When do we look for it?
Who determines it?
Funders
Proposal reviewers
PIs
Evaluators
Stakeholders
The Field
What constitutes it?
Design
Approach
Methods
Questions
Competence
Utility
Who contributes to it?
PIs, evalutors, funders, teams, etc.
At the beginning, middle and end!

6. How do I know it when I see it?
 NEIL

7. Elements of evaluation quality
Evaluators
Evaluation
AEA Evaluator Competencies
Visitor Studies Association Evaluator Competencies
AEA Guiding Principles
Program Evaluation Standards
AEA Cultural Competence Statement
Funder Guidance
(NIH/SEPA, NSF, etc.)

8. Evaluators
AEA Evaluator Competencies, 2018; VSA Evaluator Competencies, 2008

9. Evaluators – Competencies
VSA Evaluator Competencies
AEA Evaluator Competencies
A. Principles and Practices of Visitor Studies
1. Professional Practice
B. Principles and Practices of Informal Learning Environments
2. Methodology
C. Knowledge of and Practices with Social Science Research and Evaluation Methods and Analysis
3. Context
D. Business Practices, Project Planning, and Resource Management
4. Planning and Management
E. Professional Commitment
5. Interpersonal

10. AEA Ethical Guiding Principles, 2018
Evaluators
AEA Ethical Guiding Principles, 2018

11. Evaluators – Ethical Guiding Principles
Systematic Inquiry
Evaluators conduct data-based inquiries that are thorough, methodical, and contextually relevant. 
Competence
Evaluators provide skilled professional services to stakeholders. 
Integrity
Evaluators behave with honesty and transparency in order to ensure the integrity of the evaluation. 
Respect for People
Evaluators honor the dignity, well-being, and self-worth of individuals and acknowledge the influence of culture within and across groups. 
Common Good and Equity
Evaluators strive to contribute to the common good and advancement of an equitable and just society. 

12. Evaluation The Program Evaluation Standards

13. Evaluation
Program Evaluation Standards
Utility The utility standards are intended to increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs. 
Feasibility The Feasibility standards are intended to increase evaluation effectiveness and efficiency. 
Propriety The propriety standards support what is proper, fair, legal, right and just in evaluations. 
Accuracy The accuracy standards are intended to increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality. 
Accountability The evaluation accountability standards encourage adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products. 
Also methods and rigor!!

14. Evaluation AEA Public Statement on
Cultural Competence in Evaluation, 2011

15. Culturally Responsive Evaluation
Essential Practices for Cultural Competence
Acknowledge the complexity of cultural identity
Recognize the dynamics of power
Recognize and eliminate bias in language
Employ culturally appropriate methods

16. When might we look for quality?
Design
Project/context understanding
Questions
Methods
Management
Data Collection
Sources
Instruments
Analyses
Analytic methods
Interpretation
Reporting
Annual/progress reporting
Strategic/final reporting

17. Leslie’s Tips for Ensuring Evaluation Quality
Communicate early and often
It’s a relationship – approach it that way!
Plan with use in mind
Focus on learning and improving, measuring more than just outcomes
Outcomes are humans, too!
Context and culture matter – keep them front and center
Be flexible and responsive
Look for quality in all phases of evaluation

18. Where can I learn more?
 NEIL

19. Resources!
Checklist for the Program Evaluation Standards American Evaluation Association Visitor Studies Association Center for the Advancement of Informal Science Center for Culturally Responsive Evaluation and Assessment
Hold up the NDE volume and mention authors!

20. Thank you
Leslie Goodyear