Elements of evaluation quality: questions, answers, and resources Leslie Goodyear, PhD Principal Evaluation Director, EDC NIH SciEd 2019: Annual Conference for NIH Science Education Projects April 22, 2019

Intro and mention of expertise in the room. Quick hands-up activity – who’s an evaluator? Who’s not?

questions  NEIL What is it, who should be concerned about it, and what do I need to know?

What is evaluation? First, let’s talk about evaluation – what is it? The systematic investigation of the quality of programs, projects…and/or their components, together or singly For purposes of decision making, judgments, conclusions, findings, new knowledge, organizational development, and capacity building in response to the identified needs of identified stakeholders Leading to improvement and/or accountability… Ultimately contributing to organizational or social value.

What is evaluation quality? Who determines it? What constitutes it? Who contributes to it? When do we look for it? Who determines it? Funders Proposal reviewers PIs Evaluators Stakeholders The Field What constitutes it? Design Approach Methods Questions Competence Utility Who contributes to it? PIs, evalutors, funders, teams, etc. At the beginning, middle and end!

How do I know it when I see it?  NEIL

Elements of evaluation quality Evaluators Evaluation AEA Evaluator Competencies Visitor Studies Association Evaluator Competencies AEA Guiding Principles Program Evaluation Standards AEA Cultural Competence Statement Funder Guidance (NIH/SEPA, NSF, etc.)

Evaluators AEA Evaluator Competencies, 2018; VSA Evaluator Competencies, 2008

Evaluators – Competencies VSA Evaluator Competencies AEA Evaluator Competencies A. Principles and Practices of Visitor Studies 1. Professional Practice B. Principles and Practices of Informal Learning Environments 2. Methodology C. Knowledge of and Practices with Social Science Research and Evaluation Methods and Analysis 3. Context D. Business Practices, Project Planning, and Resource Management 4. Planning and Management E. Professional Commitment 5. Interpersonal

AEA Ethical Guiding Principles, 2018 Evaluators AEA Ethical Guiding Principles, 2018

Evaluators – Ethical Guiding Principles Systematic Inquiry Evaluators conduct data-based inquiries that are thorough, methodical, and contextually relevant.  Competence Evaluators provide skilled professional services to stakeholders.  Integrity Evaluators behave with honesty and transparency in order to ensure the integrity of the evaluation.  Respect for People Evaluators honor the dignity, well-being, and self-worth of individuals and acknowledge the influence of culture within and across groups.  Common Good and Equity Evaluators strive to contribute to the common good and advancement of an equitable and just society. 

Evaluation The Program Evaluation Standards

Evaluation Program Evaluation Standards Utility The utility standards are intended to increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs.  Feasibility The Feasibility standards are intended to increase evaluation effectiveness and efficiency.  Propriety The propriety standards support what is proper, fair, legal, right and just in evaluations.  Accuracy The accuracy standards are intended to increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality.  Accountability The evaluation accountability standards encourage adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products.  Also methods and rigor!!

Evaluation AEA Public Statement on Cultural Competence in Evaluation, 2011

Culturally Responsive Evaluation Essential Practices for Cultural Competence Acknowledge the complexity of cultural identity Recognize the dynamics of power Recognize and eliminate bias in language Employ culturally appropriate methods

When might we look for quality? Design Project/context understanding Questions Methods Management Data Collection Sources Instruments Analyses Analytic methods Interpretation Reporting Annual/progress reporting Strategic/final reporting

Leslie’s Tips for Ensuring Evaluation Quality Communicate early and often It’s a relationship – approach it that way! Plan with use in mind Focus on learning and improving, measuring more than just outcomes Outcomes are humans, too! Context and culture matter – keep them front and center Be flexible and responsive Look for quality in all phases of evaluation

Where can I learn more?  NEIL

Resources! Checklist for the Program Evaluation Standards https://wmich.edu/evaluation/checklists American Evaluation Association http://www.eval.org/ Visitor Studies Association https://www.visitorstudies.org/ Center for the Advancement of Informal Science http://www.informalscience.org/ Center for Culturally Responsive Evaluation and Assessment https://crea.education.illinois.edu/ Hold up the NDE volume and mention authors!

Thank you Leslie Goodyear lgoodyear@edc.org