1. Social Justice Amidst Standards and Accountability
Donna M. Mertens, Keynote
Independent Consultant
Minnesota Evaluation Studies Institute, MESI Spring Training
Minneapolis MN
March 2015

2. Why this topic? Why now? Social Justice Amidst Standards and
Accountability
Mertens Keynote MN MESI March 2015
3/11/2015

3. 2015 International Year of Evaluation
EvalPartners, UNICEF, UN Women, and the International Organization for Cooperation in Evaluation
Peer 2 Peer Initiative – gender and equity focused evaluation
Mertens Keynote MN MESI March 2015
3/11/2015

4. Need for Social Justice Link to Evaluation: Pervasiveness of racism and other isms
Mertens Keynote MN MESI March 2015
3/11/2015

5. Need for a Social Justice Lens
Marie Battiste (2000): “…society is sorely in need of what Aboriginal knowledge has to offer” Chilisa (2005) “it is an issue of life and death” Brookes (2006) Failure to include racism as a potential contributor to disparities
Mertens Keynote MN MESI March 2015
3/11/2015

6. My hypothesis:
If we begin by prioritizing social justice and human rights
And we appropriately involve community members in the evaluation process,
Then we will increase the probability of social transformation as a result of our evaluation.
Mertens Keynote MN MESI March 2015
3/11/2015

7. What does our evidence lack in credibility from a social justice perspective?
Mertens Keynote MN MESI March 2015
3/11/2015

8. Social Justice Standards Accountability
Mertens Keynote MN MESI March 2015
3/11/2015

9. Accountability Question
Did “it” work?
Should we keep spending our money on “it”?
Mertens Keynote MN MESI March 2015
3/11/2015

10. Social Justice Questions
Was “it” the right thing?
Was “it” chosen and/or developed and implemented in culturally responsive ways?
Were contextual issues of culture, race/ethnicity, gender, disability, deafness, religion, language, immigrant or refugee status, age or other dimensions of diversity used as a basis for discrimination and oppression addressed?
How were issues of power addressed?
Do we want to continue to spend money on things that don’t work?
Mertens Keynote MN MESI March 2015
3/11/2015

11. Resources for Evaluators
AEA Guiding Principles
AEA Public Statement on Cultural Competency
Joint Committee Program Evaluation Standards
Evaluators who have dedicated their lives towards furthering social justice
Communities who want social justice
Mertens Keynote MN MESI March 2015
3/11/2015

12. Good news; bad news for social justice in evaluation
1.2% (n=22) in Harner’s (2014) survey of evaluators identified their theoretical frame as Social Justice
69% (n=819) of 1,187 evaluators either strongly or somewhat agreed with this statement:
Evaluation should focus on bringing about social justice.
Mertens Keynote MN MESI March 2015
3/11/2015

13. AEA Guiding Principles (2004): Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for General and Public Welfare
To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence. Cultural competence would be reflected in evaluators seeking awareness of their own culturally-based assumptions, their understanding of the worldviews of culturally-different participants and stakeholders in the evaluation, and the use of appropriate evaluation strategies and skills in working with culturally different groups. Diversity may be in terms of race, ethnicity, gender, religion, socio-economics, or other factors pertinent to the evaluation context. (Competence B2)
Mertens Keynote MN MESI March 2015
3/11/2015

14. Mertens Keynote MN MESI March 2015
AEA Guiding Principle: Cultural Competence Linkage with Transformative Axiology and Epistemology
To ensure recognition, accurate interpretation, and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence (American Evaluation Association, Public Statement on Cultural Competence in Evaluation, 2011)
Mertens Keynote MN MESI March 2015
3/11/2015

15. What is cultural competence?
Cultural competence is not a state at which one arrives; rather, it is a process of learning, unlearning, and relearning.
It is a sensibility cultivated throughout a lifetime.
Cultural competence requires awareness of self, reflection on one’s own cultural position, awareness of others’ positions, and the ability to interact genuinely and respectfully with others.
Culturally competent evaluators refrain from assuming they fully understand the perspectives of stakeholders whose backgrounds differ from their own.
Cultural competence is context dependent. (AEA 2011)
Mertens Keynote MN MESI March 2015
3/11/2015

16. Evaluation and Validity and Cultural Competence
accurately and respectfully reflect the life experiences and perspectives of program participants in their evaluations.
establish relationships that support trustworthy communication among all participants in the evaluation process.
draw upon culturally relevant, and in some cases culturally specific, theory in the design of the evaluation and the interpretation of findings.
select and implement design options and measurement strategies in ways that are compatible with the cultural context of the study. (AEA 2004, p. 6)
Mertens Keynote MN MESI March 2015
3/11/2015

17. Recognize the dynamics of power
Cultural groupings are ascribed differential status and power, with some holding privilege that they may not be aware of and some being relegated to the status of “other.” For example, language dialect and accent can be used to determine the status, privilege, and access to resources of groups. (AEA, 2004, p. 7)
Mertens Keynote MN MESI March 2015
3/11/2015

18. Standards for Good Evaluation (Yarborough et al. 2011)
Utility: evaluator credibility, attention to stakeholders, negotiated purposes, explicit purposes
Feasibility: management, practical, contextual viability (recognize, monitor, and balance the cultural and political interests and needs of individuals and groups)
Propriety: responsive and inclusive, formal agreements, human rights and respect, clarity and fairness, transparency and disclosure.
Accuracy: valid, reliable, explicit program and context descriptions, sound designs and analysis
Evaluation accountability: meta-evaluation
Mertens Keynote MN MESI March 2015
3/11/2015

19. Accountability and Social Justice
Code word for RCT?
Accountable to whom?
Ways to strengthen arguments about accountability are not limited to RCTs
Theoretical streams in evaluation provide options for demonstrating accountability
Mertens Keynote MN MESI March 2015
3/11/2015

20. Mertens Keynote MN MESI March 2015
Evaluation Theory and Social Justice – Alkin’s Tree: Methods, Use & Values Branches
Use
Methods
Values
3/11/2015
Mertens Keynote MN MESI March 2015

21. Mertens & Wilson 2012: Methods, Use, Values, & Social Justice Branches
3/11/2015
Mertens Keynote MN MESI March 2015

22. Metaphor: Tree or Water?
Mertens Keynote MN MESI March 2015
3/11/2015

23. Paradigms & Branches Paradigm Branch Post-positivist Methods
Constructivist
Values
Transformative
Social Justice
Pragmatic
Use
Mertens Keynote MN MESI March 2015
3/11/2015

24. Transformative Theories
Feminist theories
Critical Race Theories
Critical Theories
Human Rights Theories
Disability Rights Theories
Transformative Participatory Action theories
Indigenous theories
Deafness rights theories
Queer theories
Mertens Keynote MN MESI March 2015
3/11/2015

25. Transformative Paradigm
Philosophical
Assumptions
Axiology
Respect for cultural norms; support for human rights and social justice; reciprocity
Ontology
Issues of power & critical interrogation of multiple realities: social, political, cultural, economic, race/ethnic, gender, age, religion and disability values to unmask those that sustain an oppressive status quo
Epistemology
Issues of power & Interactive link; knowledge is socially and historically located; trusting relationship.
Methodology
Qualitative (dialogic)/ Quantitative / Mixed Methods; Context
3/11/2015
Mertens Keynote MN MESI March 2015

26. Court Access Project Start from community Advisory board formed
Begin understandings from the vantage point of the “least privileged”
Work with an awareness of diversity in the community
Provide respectful support for engagement
Use a cyclical approach to data collection and use
Mertens Keynote MN MESI March 2015
3/11/2015

27. Example: HIV/AIDS Prevention in Botswana
Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens
3/11/2015
Mertens Keynote MN MESI March 2015

28. Transformative Cyclical Mixed Methods Design
Assemble team; read documents; engage in dialogues
Preliminary studies: youth, older men
Process eval
Pilot intervention: Observations, Interviews, Surveys
Demographic information; Surveys; Incidence data
Pretest: Knowledge, Attitude, Behavior;
Stage 1
Qual
Stage 2
Concurrent
Stage 3
Sequential
Stage 5
Stage 4
RCT
Post
tests:
Quant
Qual;
Behavior
& Policy
Change;
Transfer
To other
contexts
Mertens Keynote MN MESI March 2015
3/11/2015

29. Indigenous Contributions
Social justice as a frame for evaluation
Engagement with indigenous communities
Mertens Keynote MN MESI March 2015
3/11/2015

30. Indigenous Paradigm Chilisa (2012) Relational Axiology Wilson (2008)
Linda T. Smith (1999)
Marie Battiste (2000)
LaFrance & Crazy Horse (2009)
Relational Axiology
Relational Ontology
Relational Epistemology
Relational Methodology
(Chilisa, 2012)
Mertens Keynote MN MESI March 2015
3/11/2015

31. Ethics and Indigeneity
Connections between people, past, present, and future, all living and nonliving things (ubuntu)
Respect and reciprocity: listens, pays attention, acknowledged, and crates space for the voices and knowledge systems of Indigenous people
Contribute to a better future
Spirituality
Mertens Keynote MN MESI March 2015
3/11/2015

32. Questions What is credible evidence?
What criteria establish credibility of evidence?
What does our evidence lack in credibility?
What sources give us insight into credibility?
What is the place of social justice in credibility?
How do voices of marginalized communities enhance our understandings of credibility?
Why is it important for researchers to understand marginalized communities’ perspectives of social justice?
Mertens Keynote MN MESI March 2015
3/11/2015

33. Methodological Questions
What data collection methods can the evaluator use to become better acquainted with the various stakeholder groups?
How can the evaluator design the study so that the community members are included in decision making?
How could the use of mixed methods contribute to the quality of the study?
How can the evaluator design the methods to increase the probability of furthering social justice?
Mertens Keynote MN MESI March 2015
3/11/2015

34. Water as Metaphor
Eventually, all things merge into one, and a river runs through it. The river was cut by the world's great flood and runs over rocks from the basement of time. - Norman Fitzroy Maclean
When you put your hand in a flowing stream, you touch the last that has gone before and the first of what is still to come. - Leonardo da Vinci
My soul is full of longing For the secret of the Sea, And the heart of the great ocean Sends a thrilling pulse through me – Henry Wadsworth Longfellow
Mertens Keynote MN MESI March 2015
3/11/2015

35. Mertens Keynote MN MESI March 2015
Resources
Mertens, D. M. & Wilson, A. (2012). Program Evaluation Theory and Practice: A Comprehensive Guide. NY: Guilford.
Mertens, D. M. (2015). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 4th ed. Thousand Oaks, CA: Sage.
Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford.
Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage.
American Evaluation Association (2011). Public Statement on Cultural Competence in Evaluation. AEA.
3/11/2015
Mertens Keynote MN MESI March 2015

36. Mertens Keynote MN MESI March 2015
Contact information
Donna M. Mertens, PhD Independent Consultant
Mertens Keynote MN MESI March 2015
3/11/2015