1. Weaving Social Justice and Evaluation together
Donna M Mertens, PhD
Independent Consultant
Minneapolis MN
March 2015

2. Introduction/Expectations-
Name, position, organization (no acronyms please).
Where do you come from?
What is your experience in evaluation?
What has been your experience with social justice?
What do you want to get out of this workshop?
Mertens Minneapolis MN March 2015

3. Transformative Paradigm
Philosophical
Assumptions
Axiology
Respect for cultural norms; support for human rights and social justice; reciprocity
Ontology
Issues of power & critical interrogation of multiple realities: social, political, cultural, economic, race/ethnic, gender, age, religion and disability values to unmask those that sustain an oppressive status quo
Epistemology
Issues of power & Interactive link; knowledge is socially and historically located; trusting relationship.
Methodology
Qualitative (dialogic)/ Quantitative / Mixed Methods; Context
Mertens Minneapolis MN March 2015

4. Transformative Mixed Methods and environmental justice
Phase 1
Qual: Identify stakeholders; build relationships; analyze documents
Quant: Identify available demographic and environmental data
Phase 2
Qual: Interviews, focus groups, town meetings
Quant: Establish baseline for pollutants and health status
Phase 3
Qual: Develop interventions based on stakeholder input
Quant/Qual: Pilot test interventions & data collection
Phase 4
Quant: process assessment of implementation; pretests
Qual: observe and interview stakeholder groups during intervention
Phase 5
Quant: Post-test measures
Qual: Interviews, observations, policy planning
Mertens Minneapolis MN March 2015

5. Contextual Factors History of the nation/country/locale
Political –government, LEGISLATION, corruption
Organizational – NGOs, universities, AGENCIES
Cultural inclusion/exclusion– ethnic groups, religion, gender, disability, languages
Power structures, discrimination/oppression
Community resilience
What about our community and the relevant organizations hinders or helps us achieve project goals?
Which contextual factors have the greatest bearing on project successes or stumbling blocks?
To what extent will the project target important community and beneficiary needs?
What economic conditions, population characteristics, related projects and services, and needs and problems of the targeted population are relevant?
Mertens Minneapolis MN March 2015

6. Context in Kyrgyz Republic: Gender
Women are more likely to make a choice in favor of education, which is later transformed into lower-paying occupations. The pay in traditionally “female” occupations is 2.5 times lower than in traditionally “male” occupations, as “female” occupations are mostly funded by the state budget and have been to a lesser extent affected by the onset of the market economy.
Men have also experienced negative consequences of socio-political and economic stereotypes, with the prevalence of stereotypes that view men only “bread-winners”. Today, the suicide rate among men in Kyrgyzstan is four times higher than among women. Average life expectancy of men is only 63.5 years, women, on average, live almost nine years longer. There is a decline in the level of education of men, which can lead to more conflicts in the family and society, lack of tolerance to differences, and greater willingness to use brute force. (National Council for Sustainable Development of the Kyrgyz Republic, 2013)
Mertens Minneapolis MN March 2015

7. Understanding the Context: Capacity Building & Organizational Assessment
What is the current status in an organization with regard to a specific focus (e.g., sufficient trained personnel? Up to date practices?)
What is the current level of knowledge about evaluation, its current uses, and attitudes toward evaluation?
What is needed to support organizational leadership, increase funding, recruit staff with evaluation expertise, conduct organization-wide training, technical assistance for those undertaking evaluations, and sharing of information in terms of strategies for evaluations and use of their findings?
Mertens Minneapolis MN March 2015

8. Evaluation in this Context
What does evaluation mean here?
What does it look like when it is done?
How do people talk about it?
Who does it?
How is it done?
How is it used?
Mertens Minneapolis MN March 2015

9. EVALUATION: DEFINITION from UN
An assessment of programme activity
Systematic and impartial
Focuses on accomplishments (expected and achieved)
Considers the results chain, processes, context, and causality
In order to determine the “relevance, impact, effectiveness, efficiency and sustainability of the intervention”
Source: UNEG, 2005
Evaluation Defined
Evaluation is:
An assessment of programme activity
Systematic and impartial
Focuses on accomplishments (expected and achieved)
Considers the results chain, processes, context, and causality
Done in order to determine the “relevance, impact, effectiveness, efficiency and sustainability of the intervention
(Source: UNEG, 2005)
Key characteristics of evaluation :
Should provide evidence-based information that is credible, reliable – and very importantly USEFUL AND APPLICABLE
Enable the timely incorporation of conclusions and recommendations into the decision-making processes of the organization
Plenary discussion (10 minutes): What is not considered in this definition of evaluation?
Mertens Minneapolis MN March 2015

10. GENDER EQUALITY
Refers to the equal rights, responsibilities, and opportunities of women and men.
It implies that the interests, needs and priorities of both women and men are taken into consideration, recognizing the diversity of different groups of women and men.
It is both a human rights issue and a precondition for, as well as an indicator of, development.
Source: UNEG Guidance
Note that many of you in this audience will be experts in the areas of gender equality and women’s rights and the following slides are just reminders of the centrality of gender equality and women’s empowerment, and CEDAW, to UNIFEM’s work. [move quickly through these].
Gender equality
Refers to the equal rights, responsibilities, and opportunities of women and men.
It implies that the interests, needs and priorities of both women and men are taken into consideration, recognizing the diversity of different groups of women and men.
It is both a human rights issue and a precondition for, as well as an indicator of, development.
Source: UNEG Guidance
Mertens Minneapolis MN March 2015

11. GENDER EQUALITY and HUMAN RIGHTS RESPONSIVE EVALUATION
What do we mean by gender equality and human rights responsive evaluation?
It is an evaluation that integrates a gender equality and a human rights based approach by:
specifically assessing the extent to which the programme evaluated is guided by organizational and system-wide objectives on gender and human rights AND
Incorporating these approaches in the actual evaluation processes
GENDER EQUALITY and HUMAN RIGHTS RESPONSIVE EVALUATION
What do we mean by gender equality and human rights responsive evaluation?
It is an evaluation that integrates a gender equality and a human rights based approach by:
specifically assessing the extent to which the programme evaluated is guided by organizational and system-wide objectives on gender and human rights AND
Incorporating these approaches in the actual evaluation processes
Mertens Minneapolis MN March 2015
Module 1-2 11 11

12. Transformative EVALUATION
Recognize power relations;
Identify the structural causes of inequality and discrimination;
Determine the impacts of programmes on impoverished people and groups facing discrimination;
Involve participatory and reflective processes; and
Acknowledge human rights and equality and determine the claims of rights- holders and obligations of duty- bearers
Aim for the progressive realization of human rights and gender equality
Source: UNEG/UNSSC, Module, 2008; Mertens & Wilson, 2012
GE and HR Responsive Evaluation
Recognize power relations;
Identify the structural causes of inequality and discrimination;
Determine the impacts of programmes on impoverished people and groups facing discrimination;
Involve participatory and reflective processes; and
Acknowledge women’s human rights and gender equality and determine the claims of rights- holders and obligations of duty- bearers
Aim for the progressive realization of human rights and gender equality
Mertens Minneapolis MN March 2015
Module 1-2 12 12

13. Beliefs about Evaluation
By stakeholder groups
What makes a good evaluation?
Purpose of evaluation
Usefulness of evaluation
Approaches to evaluation
Government
Evaluators
NGOs
Donor agencies
Marginalized communities
Etc.
Mertens Minneapolis MN March 2015

14. Advocacy for Good Evaluation: Standards for Judging Evaluation
2010 Program Evaluation Competition Workshop:
Introduction to Program Evaluation
Advocacy for Good Evaluation: Standards for Judging Evaluation
Utility
Feasibility
Propriety
Accuracy
Meta-Evaluation
Mertens Minneapolis MN March 2015

15. Think about possible evaluation purposes and implied uses…..
Cynical: Because it is required by the funding agency. Possible use? Funding agency may make decisions to continue or cut funding, but probably the program will not use it to improve…
Public relations: Only positive evaluation data will be reported. Possible use: Make us look good….
Mertens Minneapolis MN March 2015

16. Multiple purposes are possible for one evaluation
Purpose of evaluation may be limited to a short time frame and may be specifically focused on one aspect of a program
Purpose of evaluation may change as the evaluand goes through different phases over a period of time (years…)
You may have several purposes in your evaluation to meet different stakeholders needs.
Mertens Minneapolis MN March 2015

17. Addressing Evaluation Purposes: Influencing the Conversation
Gain insights or determine necessary inputs (context and needs assessment)
Find areas in need of improvement or to change practices (formative)
Assess program effectiveness (summative)
Address issues of human rights and social justice (transformative)
Mertens Minneapolis MN March 2015

18. Purpose: To Gain insight/Determine Necessary Inputs
Evaluation Types:
Context
Capacity building/Organizational assessment
Needs and Assets Assessment
Relevance of current services/structure
Mertens Minneapolis MN March 2015

19. Evaluation Purpose: Improvement or Change Practices
Implementation
Responsive
Participatory
Process,
Monitoring,
Formative
Developmental
Mertens Minneapolis MN March 2015

20. Purpose: To Assess Program Effectiveness
Outcomes,
Impact,
Summative,
Cost analysis,
Policy,
Replicability/
Exportability
Sustainability
Mertens Minneapolis MN March 2015

21. Evaluation Purpose – Human Rights & Social Justice
Country Led
Critical Race Theory
Indigenous
Culturally Responsive
Disability and Deafness Rights
Feminist
Gender analysis
Transformative participatory
Mertens Minneapolis MN March 2015

22. How do you get people to agree on the purpose of the evaluation?
Assemble appropriate people (stakeholders)
Ask appropriate questions:
What are we doing right?
Is there a problem?
What has changed in the environment/organization that suggests a need for change?
What new directions are possible?
Mertens Minneapolis MN March 2015

23. Team Development: Who are your stakeholders?
Everyone impacted by results – even if they are impacted by being excluded from participation…
Project administrators/funders/policy makers;
Project implementors (staff)
Participants and nonparticipants
BUT how can you include ALL those people?
HINT: Sometimes you have to sample w/in groups to have a manageable number of stakeholders. smile
Mertens Minneapolis MN March 2015

24. Team Development
identify existing community action groups and understand the history of their efforts;
identify existing formal, informal, and potential leaders;
identify community needs and gaps in services;
identify community strengths and opportunities;
understand your target population (both needs and assets) in order to improve, build, and secure project credibility within the community; and
create a momentum for project activities by getting community input.
Mertens Minneapolis MN March 2015

25. Who is on the program team?
How reflective are team members of the targeted community?
To what extent do underrepresented groups (disaggregated) have input into decisions about what and how issues will be addressed and how the impact of the interventions will be measured?
How is resource distribution affecting the ability of stakeholders to benefit from the innovations?
Who cannot participate and why?
How can power differences be safely acknowledged and accommodated?
Mertens, 2009, p. 206, TRE
Mertens Minneapolis MN March 2015

26. How to involve stakeholders?
Establish a consultative & advisory group
Who? key representatives from national government counterparts, partners, CSOs, women’s organizations, beneficiaries, program managers, other Agencies and/ or donors
Role: develop evaluation terms of reference, decision making, providing contextual or technical expertise and/or aiding in evaluation data collection, report preparation, dissemination and use
Mertens Minneapolis MN March 2015

27. Constraints & Opportunities
Sociopolitical factors – the bridge between the stakeholders and the constraints
Identify and effectively address the affected/concerned groups
Internal info that is gathered – who needs access to that? How will it be presented?
Realistic expectations – set limits – how can you communicate with the stakeholders, and what can the evaluation really do?
How will you build confidence in you? Respect and honesty with them will improve their confidence, so be sure to show them those qualities
Mertens Minneapolis MN March 2015

28. Specific constraints Be sure to include time, money, and politics
Is the program open to being evaluated? Accepting of the evaluation?
How supportive are the leaders of the evaluation?
What is the history in the organization related to evaluation?
Consider the setting – Is your evaluation to occur in one place or spread out (e.g., the PT3 Join Together evaluation or the Nat’l Info Center – these are "virtual communities", where are their “boundaries”?)
Economics can play a role in how supportive the organization is of the evaluation
Social atmosphere of the program? OK to criticize or not?
Mertens Minneapolis MN March 2015

29. Selecting a program (project): Things to think about …
Representatives of marginalized groups are included in the decisions about which program to evaluate and how to evaluate it.
Evaluators and intended users agree on the goals, objectives, side effects, and performance criteria
Program goals and objectives are found to be realistic given the resources available
Relevant information on program performance is available
Decision-makers on the policy or operating level are willing to change the program on the basis of evaluation information.
 SCRIPT:
An evaluability assessment is conducted to determine if a program is ripe or ready for evaluation.
Bullet 1 and 2:
An evaluation of a program that lacks clearly defined or unrealistic goals, objectives and performance criteria is difficult to conduct because there are no clear standards against which to assess the achievement of intended goals, objectives and outcomes.
Bullet 3:
Performance measurement and program evaluation are complimentary. Performance measurement data are needed to conduct an evaluation. An evaluation will be richer and more meaningful and conclusive with better data. The lack of data needed to assess program performance may delay the conduct of an evaluation.
Mertens Minneapolis MN March 2015

30. What is a Logic Model?
A logic model is a diagram and text that describes/ illustrates the logical (causal) relationships among program elements and the problem to be solved, thus defining measurements of success.
Mertens Minneapolis MN March 2015

31. SCRIPT:
This slide illustrates the plight of many programs. Most people know what resources are used to support a program and they see the outcomes/results of a program, but they don’t have a clear understanding of how or why the results occurred. The piece in the middle is a mystery – this is the miracle that occurs. In the cartoon, the logic model is a tool that will help you demystify the piece in the middle. It provides a structure, framework and process for helping the user understand what happens in the middle.
Government programs historically have focused attention on activities and outputs, and assumed that these would somehow translate (the “Miracle”) into achievement of their long-term strategic goals.
In a logic model, unlike the cartoon, we want to adequately specify the program theory so we all know what has to be done (i.e. fill in the “then a miracle occurs”).
You could introduce the logic model as a recipe or a map – in both cases you start with the end in mind and then look for a tool (recipe or map) that will help chart the course to success – getting to the end in mind!
Mertens Minneapolis MN March 2015

32. Everyday example Get food Eat food Feel better H U N G R Y
In this case, we are hungry.
Our experience tells us that if we could just get some food
Then, eat the food;
Then we will not longer be hungry and we will feel better.
The real thing is that we want to feel better- the desired end result.
Mertens Minneapolis MN March 2015

33. Level I Logic Model YOUR PROGRAM RESULTS/IMPACT RESOURCES/INPUTS
What you do to achieve your long-term aims!
RESULTS/IMPACT
Why you are in business!
RESOURCES/INPUTS
What you need
to implement
your program!
Mertens Minneapolis MN March 2015

34. Cook, play, talk, laugh, hike
Every day logic model –
Family Vacation
Family Members
Drive to state park
Family members learn about each other; family bonds; family has a good time
Budget
Set up camp
Car
Cook, play, talk, laugh, hike
Camping Equipment
In this example, it is summertime and we want to take a family vacation.
We have had experience and know (our own personal research tells us) that camping is something we all enjoy doing together. So, in order to take a camping trip, we need..
If this…, then that….
Logic models involve a mental process. A logic model shows the series of connections and logical linkages that is expected to result in achievement of our goal.
INPUTS
OUTPUTS
OUTCOMES
Mertens Minneapolis MN March 2015

35. The Evaluand and the Logic Model
1. Resources and/or barriers, which potentially enable or limit program effectiveness. Enabling protective factors or resources may include funding, existing organizations, potential collaborating partners, existing organizational or interpersonal networks, staff and volunteers, time, facilities, equipment, and supplies. Limiting risk factors or barriers might include such things as attitudes, lack of resources, policies, laws, regulations, and geography. 2. Activities are the processes, techniques, tools, events, technology, and actions of the planned program. These may include products – promotional materials and educational curricula; services – education and training, counseling, or health screening; and infrastructure – structure, relationships, and capacity used to bring about the desired results.
Mertens Minneapolis MN March 2015

36. Evaluand/Logic Model continued
3. Outputs are the direct results of program activities. They are usually described in terms of the size and/or scope of the services and products delivered or produced by the program. They indicate if a program was delivered to the intended audiences at the intended “dose.” A program output, for example, might be the number of classes taught, meetings held, or materials produced and distributed; program participation rates and demography; or hours of each type of service provided. 4. Outcomes are specific changes in attitudes, behaviors, knowledge, skills, status, or level of functioning expected to result from program activities and which are most often expressed at an individual level. 5. Impacts are organizational, community, and/or system level changes expected to result from program activities, which might include improved conditions, increased capacity, and/or changes in the policy arena.
Mertens Minneapolis MN March 2015

37. Mertens Minneapolis MN March 2015

38. Mertens Minneapolis MN March 2015

39. Verify Logic Model with Stakeholders
Is the level of detail sufficient to create understandings of the elements and their interrelationships?
Is the program logic complete? That is, are all the key elements accounted for?
Is the program logic theoretically sound? Do all the elements fit together logically? Are there other plausible pathways to achieving the program outcomes?
Have all the relevant external contextual factors been identified and their potential influences described?
Mertens Minneapolis MN March 2015

40. Challenging Theories of Change
Theories of change need to reflect:
Diversity
Power differentials
The real on-the-ground experience of intended staff and participants
Social or psychological theories about behavior change
Mertens Minneapolis MN March 2015

41. Example: HIV/AIDS Prevention in Botswana
Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens
Mertens Minneapolis MN March 2015

42. Evaluation Questions for Gaining Insight/Determining Needs & Assets
Needs assessment to determine the needs for the targeted stakeholders –
What is the current status; what needs are unmet or inadequately met?
What would be helpful?
What resources are available?
What activities seem most likely to help reach the evaluand’s goals?
Mertens Minneapolis MN March 2015

43. Evaluation Purpose: Improvement or Change Practices – Implementation Questions
What are the critical components/activities of this project (both explicit and implicit)?
How do these components connect to the goals and intended outcomes for this project?
What aspects of the implementation process are facilitating success or acting as stumbling blocks for the project?
How is the program being implemented and how does that compare to the initial plan for implementation?
Mertens Minneapolis MN March 2015

44. Evaluation Purpose – Assess program effectiveness –Impact Evaluation Questions
To what extent were the objectives achieved or likely to be achieved?
What were the major factors influencing the achievement or non-achievement of the objectives?
What has happened as a result of the project or program?
What real difference has the activity made to the beneficiaries?
How many people were affected?
(Impact evaluation is usually interpreted as longer term effects of the intervention.)
Mertens Minneapolis MN March 2015

45. Evaluation Purpose – Human Rights & Social Justice –Gender Analysis Questions
Who will benefit and who will lose from this project in terms of gender? Short-term and long-term?
To what extent were women involved in the development of the program?
Which organizations from government and civil society were included?
How does this project challenge existing gender divisions in terms of labor, tasks, responsibilities, opportunities, access to and control over resources? What are the risks of backlash if changes are made?
Mertens Minneapolis MN March 2015

46. TRANSFORMATIVE EVALUATION: Rigor
Emphasizes Human Rights and Social justice
Analyses asymmetric power relations
Advocates culturally competent relations between the evaluator and community members
Employs culturally appropriate mixed methods tied to social action
Applies feminist theory, critical race theory, postcolonial and indigenous theories
SOCIAL AGENDA AND TRANSFORMATIVE EVALUATION APPROACHES
Emphasizes Human Rights and Social justice
Analyses Asymmetric power relations
Advocates culturally competent relations between the evaluator and community members
Employs culturally appropriate mixed methods tied to social action
Applies feminist theory, critical race theory, postcolonial and indigenous theories
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
Mertens Minneapolis MN March 2015
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.

47. Qualitative Quantitative Concurrent Design
Quantitative and Qualitative occur more or less simultaneously
Quantitative
Qualitative
Sequential Design: Quantitative Followed by Qualitative OR
Sequential Design: Qualitative Followed by Quantitative
Qualitative
Quantitative
Mertens Minneapolis MN March 2015
Mertens, 2009, TRE, p. 167

48. Transformative Mixed Methods Design
Assemble team; read documents; engage in dialogues
Preliminary studies: youth, older men
Process eval
Pilot intervention: Observations, Interviews, Surveys
Demographic information; Surveys; Incidence data
Pretest: Knowledge, Attitude, Behavior;
Stage 1
Qual
Stage 2
Concurrent
Stage 3
Sequential
Stage 4
Post
tests:
Quant
Qual;
Behavior
& Policy
Change;
Transfer
To other
contexts
Mertens Minneapolis MN March 2015

49. TRANSFORMATIVE EVALUATION: Rigor
Emphasizes Human Rights and Social justice
Analyses asymmetric power relations
Advocates culturally competent relations between the evaluator and community members
Employs culturally appropriate mixed methods tied to social action
Applies feminist theory, critical race theory, postcolonial and indigenous theories
SOCIAL AGENDA AND TRANSFORMATIVE EVALUATION APPROACHES
Emphasizes Human Rights and Social justice
Analyses Asymmetric power relations
Advocates culturally competent relations between the evaluator and community members
Employs culturally appropriate mixed methods tied to social action
Applies feminist theory, critical race theory, postcolonial and indigenous theories
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
Mertens Minneapolis MN March 2015
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.

50. Utilization
The value of sharing information throughout the course of the research or evaluation study is emphasized to facilitate midcourse corrections if an intervention is not moving toward the desired goal.
Planning for utilization is essential during the initial design of the study; the topic of the study must be presented to participants in order to ensure that the data are gathered and disseminated in a way that they can be used to achieve the goals of social change and social justice.
Policy analysis and advocacy are avenues to social change from the perspective of grass-roots organizations.
Mertens Minneapolis MN March 2015

51. Questions: Data Reporting & Use
Who would you include in the preparation and dissemination of findings?
How would you include members of the community in this process?
What challenges might you anticipate encountering at this stage of your study?
How would you deal with power differences among those involved in these activities?
What is your thinking about ownership of the data and how would you handle this issue?
What aspects of the transformative paradigm are most salient at this point in the study?
Mertens Minneapolis MN March 2015

52. Reporting Formats Written formats: voice and privilege
Focus groups & interviews
Visual presentations
Drawing pictures
Photos, videos, slide shows
Ethnodrama
Web-based
Community-based
Mertens Minneapolis MN March 2015

53. Mixed Methods: Capture complexity, address social justice, and ethics
Conclusions
Courage
Mixed Methods: Capture complexity, address social justice, and ethics
Teams
Capacity development
Mertens Minneapolis MN March 2015

54. Mertens Minneapolis MN March 2015
Agenda for Action
How will you modify your own evaluation approach to incorporate ideas from this presentation?
Mertens Minneapolis MN March 2015

55. Mertens Minneapolis MN March 2015
Resources
Mertens, D. M. & Wilson, A. (2012). Program Evaluation Theory and Practice: A Comprehensive Guide. NY: Guilford.
Mertens, D. M. (2015). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 4th ed. Thousand Oaks, CA: Sage.
Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford.
Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage.
American Evaluation Association (2011). Public Statement on Cultural Competence in Evaluation. AEA.
Mertens Minneapolis MN March 2015

56. Mertens Minneapolis MN March 2015
Contact information
Donna M. Mertens, PhD Independent Consultant
Mertens Minneapolis MN March 2015