EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

Agenda American Evaluation Association (AEA) conference experience(s) Improvement- and accountability- oriented evaluation approaches Social agenda and advocacy evaluation approaches Eclectic approaches

NOTE: If we do not get through all of this material tonight we will continue next week…

AEA experience

What was your most ‘profound’ experience at the conference? What did you learn (or not learn that you expected to)? What did you like most? What did you like least?

Improvement- and accountability-oriented approaches

(Ideally) are expansive and cover all aspects related to merit, worth, and significance of a program or other type of evaluand Intended to inform program improvement and/or decision making (i.e., accountability) Extremely difficult to implement in practice – Stufflebeam – Scriven

Approach 15: Decision- and accountability-oriented studies Advance organizers – Relevant decision makers, decisions to be made, accountability requirements, etc.. Purposes – Decision making—whether formative or summative Sources of questions – Stakeholders (typically limited) Questions – How can the evaluand be improved? – Does the evaluand meet accountability requirements?

Methods – Any relevant method for determining how to improve a program or provide information regarding accountability Pioneers – Lee Cronbach, Daniel Stufflebeam Use considerations – Generally directed toward program management and staff Strengths – Scope Weaknesses – Emphasis on program management and staff

Approach 16: Consumer- oriented studies Advance organizers – Complex conceptions of ‘values’ (i.e., criteria) Purposes – Typically, but not always, summative – Extent to which consumers’ needs are met Sources of questions – Relevant sources of values (see Scriven’s Key Evaluation Checklist [KEC]) Questions – How well does the evaluand meet consumers’ needs?

Methods – Any relevant method for determining how well consumers’ needs have been met Pioneers – Michael Scriven Use considerations – Emphasis is on instrumental use Strengths – Independence Weaknesses – Independence

Approach 17: Accreditation and certification Advance organizers – Accreditation or certification requirements Purposes – Whether accreditation or certification requirements are met Sources of questions – Accreditation or certification requirements Questions – Have accreditation or certification requirements been met?

Methods – Any relevant method for determining how well consumers’ needs have been met Pioneers – Various Use considerations – Accordance to accepted standards Strengths – For making informed judgments Weaknesses – Emphasis on inputs and processes

Social agenda and advocacy approaches

Predominately aimed at increasing social justice Giving power to marginalized/disenfranchised groups Tend to favor ‘qualitative’ methods and a constructivist/constructionist perspective Stress cultural pluralism, relativism, and multiple realities Only a small emphasis on determining merit, worth, or significance

Approach 18: Responsive or stakeholder-centered evaluation Advance organizers – Stakeholders’ concerns Purposes – Varies widely, depending on stakeholders’ needs Sources of questions – Stakeholders Questions – What has the program achieved? – How well has the program been implemented? – What do experts ‘think’ about the program?

Methods – Most often qualitative Document analysis Observations Interviews Pioneers – Robert Stake Use considerations – Primary stakeholders Strengths – Intended to address stakeholders’ concerns Weaknesses – External credibility

Approach 19: Constructivist evaluation Advance organizers – Rejects the notion that merit, worth, and significance can be ‘objectively’ determined Purposes – To ‘construct’ stakeholders’ experiences Sources of questions – Pluralistic values of stakeholders Questions – Emergent

Methods – Predominately qualitative Pioneers – Evon Guba, Yvonne Lincoln Use considerations – Often conflicting accounts and no definitive judgments Strengths – Stakeholder involvement Weaknesses – Time, costs, and—most importantly— relativist results

Approach 20: Deliberative democratic evaluation Advance organizers – Democratic participation – Dialogue – Deliberation Purposes – Democratic participation Sources of questions – Stakeholders/evaluator(s) Questions – What is the merit and/or worth of the program?

Methods – Any relevant methods for gathering evidence Pioneers – Ernie House Use considerations – Should represent interests of all relevant stakeholders Strengths – Democratic participation Weaknesses – Ambitious demands required to execute the approach

Approach 21: Transformative evaluation Advance organizers – Social justice and equity Purposes – Recognizing ‘situational’ nature of knowledge claims Sources of questions – Least advantaged groups of stakeholders Questions – None that can be identified in advance

Methods – Generally mixed-method Pioneers – Donna Mertens Use considerations – Inclusion of marginalized stakeholders Strengths – Emphasis on social justice Weaknesses – External credibility

Eclectic evaluation approaches

No particular philosophical or methodological orientation Draw from various evaluation concepts and methods to secure ‘useful’ evaluation findings Tend to invoke the Program Evaluation Standards

Approach 22: Utilization- focused evaluation Advance organizers – Potential users and uses Purposes – Information necessary for intended uses by intended users Sources of questions – Intended users, often determined through situational analysis Questions – Specific questions articulated by intended users

Methods – Any relevant method for addressing intended users questions Pioneers – Michael Patton Use considerations – Situational, and tailored to intended users intended uses Strengths – Maximizes evaluation impact Weaknesses – Too much user input and involvement

Approach 23: Participatory evaluation Advance organizers – Promoting buy-in and use Purposes – Tends to be directed toward program improvement Sources of questions – Intended users and other stakeholders Questions – Specific questions articulated by intended users and other stakeholders

Methods – Any relevant method for addressing intended user and stakeholder questions Pioneers – Brad Cousins Use considerations – Situational, and tailored to intended users intended uses Strengths – User-friendly Weaknesses – Sometimes, poor technical quality