Social Justice Amidst Standards and Accountability Donna M. Mertens, Keynote Independent Consultant Minnesota Evaluation Studies Institute, MESI Spring Training Minneapolis MN March 2015

Why this topic? Why now? Social Justice Amidst Standards and Accountability Mertens Keynote MN MESI March 2015 3/11/2015

2015 International Year of Evaluation EvalPartners, UNICEF, UN Women, and the International Organization for Cooperation in Evaluation Peer 2 Peer Initiative – gender and equity focused evaluation Mertens Keynote MN MESI March 2015 3/11/2015

Need for Social Justice Link to Evaluation: Pervasiveness of racism and other isms Mertens Keynote MN MESI March 2015 3/11/2015

Need for a Social Justice Lens Marie Battiste (2000): “…society is sorely in need of what Aboriginal knowledge has to offer” Chilisa (2005) “it is an issue of life and death” Brookes (2006) Failure to include racism as a potential contributor to disparities Mertens Keynote MN MESI March 2015 3/11/2015

My hypothesis: If we begin by prioritizing social justice and human rights And we appropriately involve community members in the evaluation process, Then we will increase the probability of social transformation as a result of our evaluation. Mertens Keynote MN MESI March 2015 3/11/2015

What does our evidence lack in credibility from a social justice perspective? Mertens Keynote MN MESI March 2015 3/11/2015

Social Justice Standards Accountability Mertens Keynote MN MESI March 2015 3/11/2015

Accountability Question Did “it” work? Should we keep spending our money on “it”? Mertens Keynote MN MESI March 2015 3/11/2015

Social Justice Questions Was “it” the right thing? Was “it” chosen and/or developed and implemented in culturally responsive ways? Were contextual issues of culture, race/ethnicity, gender, disability, deafness, religion, language, immigrant or refugee status, age or other dimensions of diversity used as a basis for discrimination and oppression addressed? How were issues of power addressed? Do we want to continue to spend money on things that don’t work? Mertens Keynote MN MESI March 2015 3/11/2015

Resources for Evaluators AEA Guiding Principles AEA Public Statement on Cultural Competency Joint Committee Program Evaluation Standards Evaluators who have dedicated their lives towards furthering social justice Communities who want social justice Mertens Keynote MN MESI March 2015 3/11/2015

Good news; bad news for social justice in evaluation 1.2% (n=22) in Harner’s (2014) survey of evaluators identified their theoretical frame as Social Justice 69% (n=819) of 1,187 evaluators either strongly or somewhat agreed with this statement: Evaluation should focus on bringing about social justice. Mertens Keynote MN MESI March 2015 3/11/2015

AEA Guiding Principles (2004): Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for General and Public Welfare To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence. Cultural competence would be reflected in evaluators seeking awareness of their own culturally-based assumptions, their understanding of the worldviews of culturally-different participants and stakeholders in the evaluation, and the use of appropriate evaluation strategies and skills in working with culturally different groups. Diversity may be in terms of race, ethnicity, gender, religion, socio-economics, or other factors pertinent to the evaluation context. (Competence B2) Mertens Keynote MN MESI March 2015 3/11/2015

Mertens Keynote MN MESI March 2015 AEA Guiding Principle: Cultural Competence Linkage with Transformative Axiology and Epistemology To ensure recognition, accurate interpretation, and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence (American Evaluation Association, Public Statement on Cultural Competence in Evaluation, 2011) Mertens Keynote MN MESI March 2015 3/11/2015

What is cultural competence? Cultural competence is not a state at which one arrives; rather, it is a process of learning, unlearning, and relearning. It is a sensibility cultivated throughout a lifetime. Cultural competence requires awareness of self, reflection on one’s own cultural position, awareness of others’ positions, and the ability to interact genuinely and respectfully with others. Culturally competent evaluators refrain from assuming they fully understand the perspectives of stakeholders whose backgrounds differ from their own. Cultural competence is context dependent. (AEA 2011) Mertens Keynote MN MESI March 2015 3/11/2015

Evaluation and Validity and Cultural Competence accurately and respectfully reflect the life experiences and perspectives of program participants in their evaluations. establish relationships that support trustworthy communication among all participants in the evaluation process. draw upon culturally relevant, and in some cases culturally specific, theory in the design of the evaluation and the interpretation of findings. select and implement design options and measurement strategies in ways that are compatible with the cultural context of the study. (AEA 2004, p. 6) Mertens Keynote MN MESI March 2015 3/11/2015

Recognize the dynamics of power Cultural groupings are ascribed differential status and power, with some holding privilege that they may not be aware of and some being relegated to the status of “other.” For example, language dialect and accent can be used to determine the status, privilege, and access to resources of groups. (AEA, 2004, p. 7) Mertens Keynote MN MESI March 2015 3/11/2015

Standards for Good Evaluation (Yarborough et al. 2011) Utility: evaluator credibility, attention to stakeholders, negotiated purposes, explicit purposes Feasibility: management, practical, contextual viability (recognize, monitor, and balance the cultural and political interests and needs of individuals and groups) Propriety: responsive and inclusive, formal agreements, human rights and respect, clarity and fairness, transparency and disclosure. Accuracy: valid, reliable, explicit program and context descriptions, sound designs and analysis Evaluation accountability: meta-evaluation Mertens Keynote MN MESI March 2015 3/11/2015

Accountability and Social Justice Code word for RCT? Accountable to whom? Ways to strengthen arguments about accountability are not limited to RCTs Theoretical streams in evaluation provide options for demonstrating accountability Mertens Keynote MN MESI March 2015 3/11/2015

Mertens Keynote MN MESI March 2015 Evaluation Theory and Social Justice – Alkin’s Tree: Methods, Use & Values Branches Use Methods Values 3/11/2015 Mertens Keynote MN MESI March 2015

Mertens & Wilson 2012: Methods, Use, Values, & Social Justice Branches 3/11/2015 Mertens Keynote MN MESI March 2015

Metaphor: Tree or Water? Mertens Keynote MN MESI March 2015 3/11/2015

Paradigms & Branches Paradigm Branch Post-positivist Methods Constructivist Values Transformative Social Justice Pragmatic Use Mertens Keynote MN MESI March 2015 3/11/2015

Transformative Theories Feminist theories Critical Race Theories Critical Theories Human Rights Theories Disability Rights Theories Transformative Participatory Action theories Indigenous theories Deafness rights theories Queer theories Mertens Keynote MN MESI March 2015 3/11/2015

Transformative Paradigm Philosophical Assumptions Axiology Respect for cultural norms; support for human rights and social justice; reciprocity Ontology Issues of power & critical interrogation of multiple realities: social, political, cultural, economic, race/ethnic, gender, age, religion and disability values to unmask those that sustain an oppressive status quo Epistemology Issues of power & Interactive link; knowledge is socially and historically located; trusting relationship. Methodology Qualitative (dialogic)/ Quantitative / Mixed Methods; Context 3/11/2015 Mertens Keynote MN MESI March 2015

Court Access Project Start from community Advisory board formed Begin understandings from the vantage point of the “least privileged” Work with an awareness of diversity in the community Provide respectful support for engagement Use a cyclical approach to data collection and use Mertens Keynote MN MESI March 2015 3/11/2015

Example: HIV/AIDS Prevention in Botswana Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens 3/11/2015 Mertens Keynote MN MESI March 2015

Transformative Cyclical Mixed Methods Design Assemble team; read documents; engage in dialogues Preliminary studies: youth, older men Process eval Pilot intervention: Observations, Interviews, Surveys Demographic information; Surveys; Incidence data Pretest: Knowledge, Attitude, Behavior; Stage 1 Qual Stage 2 Concurrent Stage 3 Sequential Stage 5 Stage 4 RCT Post tests: Quant Qual; Behavior & Policy Change; Transfer To other contexts Mertens Keynote MN MESI March 2015 3/11/2015

Indigenous Contributions Social justice as a frame for evaluation Engagement with indigenous communities Mertens Keynote MN MESI March 2015 3/11/2015

Indigenous Paradigm Chilisa (2012) Relational Axiology Wilson (2008) Linda T. Smith (1999) Marie Battiste (2000) LaFrance & Crazy Horse (2009) Relational Axiology Relational Ontology Relational Epistemology Relational Methodology (Chilisa, 2012) Mertens Keynote MN MESI March 2015 3/11/2015

Ethics and Indigeneity Connections between people, past, present, and future, all living and nonliving things (ubuntu) Respect and reciprocity: listens, pays attention, acknowledged, and crates space for the voices and knowledge systems of Indigenous people Contribute to a better future Spirituality Mertens Keynote MN MESI March 2015 3/11/2015

Questions What is credible evidence? What criteria establish credibility of evidence? What does our evidence lack in credibility? What sources give us insight into credibility? What is the place of social justice in credibility? How do voices of marginalized communities enhance our understandings of credibility? Why is it important for researchers to understand marginalized communities’ perspectives of social justice? Mertens Keynote MN MESI March 2015 3/11/2015

Methodological Questions What data collection methods can the evaluator use to become better acquainted with the various stakeholder groups? How can the evaluator design the study so that the community members are included in decision making? How could the use of mixed methods contribute to the quality of the study? How can the evaluator design the methods to increase the probability of furthering social justice? Mertens Keynote MN MESI March 2015 3/11/2015

Water as Metaphor Eventually, all things merge into one, and a river runs through it. The river was cut by the world's great flood and runs over rocks from the basement of time. - Norman Fitzroy Maclean When you put your hand in a flowing stream, you touch the last that has gone before and the first of what is still to come. - Leonardo da Vinci My soul is full of longing For the secret of the Sea, And the heart of the great ocean Sends a thrilling pulse through me – Henry Wadsworth Longfellow http://www.finestquotes.com/select_quote-category-Water-page-1.htm#ixzz2968zeXR9 Mertens Keynote MN MESI March 2015 3/11/2015

Mertens Keynote MN MESI March 2015 Resources Mertens, D. M. & Wilson, A. (2012). Program Evaluation Theory and Practice: A Comprehensive Guide. NY: Guilford. Mertens, D. M. (2015). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 4th ed. Thousand Oaks, CA: Sage. Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford. Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage. American Evaluation Association (2011). Public Statement on Cultural Competence in Evaluation. AEA. 3/11/2015 Mertens Keynote MN MESI March 2015

Mertens Keynote MN MESI March 2015 Contact information Donna M. Mertens, PhD Independent Consultant Donna.Mertens@Gallaudet.edu Mertens Keynote MN MESI March 2015 3/11/2015