1. `Show me your impact`: Evaluating historic memory & racism in Guatemala Colleen Duggan International Development Research Centre (IDRC), Canada American Evaluation Association Conference, Nov. 2009
Presentation today part of a larger paper looking at the challenges and opportunities for evaluating Transitional Justice
Today: Focussing on case study portion, in order to illustrate some of the major evaluation issues already highlighted by the other presenters
Give an overview of the case study itself, challenges encountered in monitoring and evaluating this particular development intervention – what we learned and how this learning has been incorporated into program improvement.

2. Case Context: Guatemala`s conflict, historic memory and (re) education
Signing of Peace Agreements in 1996; Conflict legacy: Guatemalan internal armed left about 2% of the national population dead or disappeared – 83% of who were indigenous.
In 1999, Guatemala`s Truth Commission concluded that the Guatemalan state had committed acts of ethnic genocide against the indigenous population
Guatemala faces the challenge of dealing with its past, healing a society which allowed the “unthinkable” to happen.
-Truth Commission also discussed need for initiatives for historic memory AND the reform of primary, secondary and university level education to include instruction on the causes, development and consequences of the war.
-History education only figured fleetingly in the Guatemalan Truth Commission report - BUT the transformation of Guatemala`s education system was clearly recognized as a priority in the Peace Agreements. See Agreement on Identity and Rights of Indigenous Peoples (1995).
-Although most Guatemalans recognize that racism is a problem, constructing a nation in which ethnic diversity is celebrated and everyone is a citizen in the fullest sense of the word is a task for many generations to come.

3. CIRMA-IDRC Partnership
: Moving research into action for social change
National Campaign for Interethnic dialogue, Our Diversity is our Strength!
Museum expo
Group dialogue
University teaching series
-In this context that the Centro de Investigaciones Regionales de Meso América (CIRMA,) a research centre and library founded in the midst of the armed conflict in 1980, decided to develop research and educational strategies around the issue of racism – seen as a central cause of the conflict - as a contribution to the social reconstruction of Guatemala in the post-war period.
-IDRC`s relationship with CIRMA pre-dated the release of the Truth Commission Report. Between 1998 and 2003, IDRC, had supported CIRMA to produce an exhaustive multi-volume ethnographic study of the history and current tendencies of ethnic relations in Guatemala.
- In 2003 CIRMA began discussing with IDRC and other donors the potential for using this research to reconstruct a new Guatemalan narrative around race relations and the history of the conflict. CIRMA`s research formed the basis for an ambitious nation-wide effort to stimulate public reflection and dialogue.
-National Campaign for Interethnic Dialogue: Our Diversity is Our Strength!, was developed on the basis of a national consultation, which detected strong interest in addressing the issue of racism in Guatemala; how to do this without exacerbating existing tensions in the politically and socially complex post-conflict, transitional context.
This challenge – how to create a mechanism for dialogue which would foster, and not hinder, intercultural reconciliation – was at the heart of the design of the Campaign launched in June 2004.
-The Campaign had three components, developed with help from international experts on social violence and reconciliation and based on learning emerging from museums of conscience and race consciousness training experiences:
o A 5,000 square foot interactive exposition, Por Qué Estamos Como Estamos?
(Why are we the way we are?) The expo presented the history and current state of interethnic relations in Guatemala in an animated and colourful way through photographic images, videos and interactive games that invite people to reflect on their own lives in Guatemalan society.
o A guided process for group dialogue on the topic: the dialogues groups were meant to accompany the expo and aimed to engage community leaders who were seen as multipliers in a discussion on how to transform the prevailing system; and
o A series of university teaching activities oriented toward designing an effective
teaching package on racism and social inequality.
This presentation and paper focuses upon the first component: MUSEUM EXPO

4. How do we know if it`s `working`? M & E Framework
Development of an M & E Framework
Outcome Mapping design
Data collection instruments:
Entry and exit questionnaires, records of attendance, weekly reports by expo museum guides, interviews of expo attendees, participant observation; media monitoring
Monitoring the Effects of Historic Memory: What we set out to do
IDRC`s Evaluation Unit offered to support CIRMA in conceptualizing and putting into place a comprehensive system for monitoring and evaluating the results, including the effects of the campaign. Monitoring peoples’ reactions to the campaign was seen as a critical first step in assessing how the campaign might contribute to changing peoples’ attitudes and behaviors towards racism.
-Through a series of peer support meetings and formal training in a planning, monitoring and evaluation methodology known as Outcome Mapping, IDRC worked with CIRMA staff in order to help them identify key boundary partners, those individuals and groups who the campaign most hoped to influence in present day Guatemala: high school teachers in public and private schools, regional university authorities, university professors in the regions and social and political leaders in NGOs
-IDRC support assisted CIRMA to put together a global monitoring and evaluation strategy that included a series of monitoring and data collection instruments drawn from Outcome Mapping and other methodologies
-Data collection instruments: entry and exit questionnaires, records of attendance at the expo, weekly reports by expo museum guides, interviews of expo attendees, participant observation and monitoring national and international press coverage of the campaign..
-Monitoring efforts were largely focused upon the `Why are we the way we are?` museum expo which was considered by CIRMA to constitute the heart of the national campaign operation.
-IDRC and CIRMA each had interests – some of them shared - on issues of learning and accountability for results around the museum expo. CIRMA wanted to use its research to catalyze a process of national reflection on issues of identity and deep social transformation.
- IDRC wanted to know if its funding was achieving its desired outcomes and was also interested to see how Outcome Mapping might be successfully grafted onto a process for transitional justice and reconciliation. - We also harbored the hope that data harvested from the monitoring and evaluation framework might be used to inform CIRMA`s continued research on inter-ethnic relations and social reconciliation in Guatemala.

5. What actually happened?
Deployed in three regions: 117,000 visitors or 1% of the population
Supported by 17 national and international donors
Reactions: Postive and negative; diverse, unpredictable
National and international interest – adapt for other contexts
Summative evaluations: 2007, 2008
Results from the first two years of the deployment of the museum expo in three regions of the country exceeded expectations, generating a wave of reactions both inside Guatemala and internationally. Between mid-2004 and mid-2006, the expo reached more than 117,000 visitors (nearly 1% of the national population). More people visited the exposition than visited Guatemala’s other 22 museums combined over the same period of time.
-Campaign received support from 17 national and international donors and from of a broad cross-section of Guatemalan society, an extremely rare achievement in this post-conflict society.
-The data gathered by the campaign indicated that the effort was received in general with high levels of acceptance and legitimacy. About 90% of the visitors to the exposition responded “positively” while 5% on either side viewed it as either too “light” or too “radical.”
-Reactions tended to be positive although a minority of the visitors were defensive and negative. The range of reactions to the expo underscore the deep complexity inherent in recasting historic memory and cultural identity
 -Awakened the interest of diverse international organizations working in the field of human rights and research and education in post-conflict societies. A number of international foundations requested presentations on the campaign or visited the expo with a view to documenting it as a model for reconciliation and educational innovation in a host of countries including Ethiopia, Eritrea, Serbia, Romania, Macedonia, Burma, Cambodia, Bolivia and Argentina.

6. What worked: Learning through M & E
Mass exposure and the creation of a neutral space to discuss a socially contentious issue
(unexpected outcome)
Young people a critical group, potential change agents or champions (validated assumption)
Teachers are critical – but could be detractors (useful program improvement)
Some parts of the expo more effective for generating cognitive change (content change)
WHAT WORKED: What was learned about historic memory and education in Guatemala?

7. What didn`t work: Challenges encountered, opportunities missed
Critical importance of defining data use
External context: Balancing power differentials
between funders and funded
Organisational context : internal power struggles
Clash of linear and systems approches in:
- M & E
- Theory of change
Illusive program theory
Falling into the `project trap`
WHAT DIDN`T WORK: CHALLENGES
-Missed some major opportunities in efforts to track and understand the effects of the museum expo as a tool for social reconciliation.
- In mid-2007, CIRMA closed the expo in order to evaluate its results and plan for its re-design and re-launch in As part of that process of reflection, IDRC commissioned an external summative evaluation of the monitoring and evaluation framework experience. While many important lessons emerged from the evaluation, I will restrict myself to sharing three major learnings:
The critical importance of defining use of findings
-.Much time was invested in working with CIRMA o define how the data and analysis that would emerge from the monitoring and evaluation framework would be used and who exactly would use it. We used a Utilization Focused Evaluation, to the design of the the M & E framework
- endeavored not to dictate any particular evaluation model, method, theory or use of findings.
-Expectation was that in guiding CIRMA through an interactive process, they would be better positioned to collect the data that the organisation would need for accountability purposes vis-à-vis its donors and for learning purposes around the effectiveness of the expo.
Unexpected outcomes:
1. Lack of agreement within CIRMA whether the framework and data collected should be used to inform decision-making for improving the operations of the expo thus informing program effectiveness and fulfilling accountability requirements with donors; or whether the data should be used to feed and inform CIRMA`s research activities. Although the data being yielded could arguably serve both purposes, the reality is that these are two fundamentally different groups of uses and users.
This lack of clarity generated ongoing organisational tensions between academic researchers involved in the project and those who held very operational responsibilities.
- Our approach to M & E inadvertently fed into and exacerbated an existing power differential within the organisation; A split was between those who wanted to use the expo some sort of research laboratory or generator of interesting data that could then inform a future research agenda within the organisation vs. those who viewed and wanted to use the expo as a dynamic instrument for igniting a transformative process of social change within individuals who viewed the expo.
-- Advocates vs. researchers OR traditional researchers vs. action researchers.
2. Absence of spaces within CIRMA for analysis of and reflection about emerging data. The daily operation of the national campaign was an all consuming activity for CIRMA and its staff. The massive amounts of rich qualitative data that was collected on the immediate effects that the expo was having on visitors`attitudes tended to be inconclusive, due to weak baseline data. Despite this problem, informal daily exchanges between staff allowed CIRMA to incorporate some changes during execution and did significantly contribute to the re-design of content and process for the re-launch of the expo in 2009.
2. Balancing power relationships
-Power differentials at two different levels: Externally, between CIRMA and the 17 national and international donors who were financing the campaign; and internally between CIRMA staff who were directly involved with the Expo and those who continued to be occupied by CIRMA`s traditional research and organization activities.
-CIRMA was forced to spend an inordinate amount of time reporting to different donors. Multiple donors with different reporting requirements called upon CIRMA to report on diverse outcomes (and impacts) with differing data.
An ongoing tension between those who valued quantitative vs. qualitative data.
-IDRC unwittingly played a part in this drama: the summative evaluation of the project commissioned by IDRC in 2008 discovered that CIRMA staff involved in the monitoring and evaluation work felt obligated to maintain an exclusive focus on Outcome Mapping and its data collection tools – at a time when additional more quantitative tools were needed for donor reporting - because IDRC was funding the M & E project.
In addition, part way through the project, the coordinator of the monitoring and evaluation work departed, taking with him his knowledge of Outcome Mapping.
- The new coordinator`s background was quantitative in nature. Although he managed to collect an impressive volume of quantitative (and largely demographic) data, opportunities to drill down to uncover qualitative considerations were lost.
-The campaign itself and the M & E project generated unexpected tensions within CIRMA.
-The campaign was like a donor magnet, reproducing within CIRMA all of the worst deformations of international aid. The management of the daily of operations of the monitoring and evaluation project was highly centralized in the two coordinators who oversaw this work.
-Despite their best efforts, the coordinators were unsuccessful in generating excitement and building ownership for monitoring activities. The museum guides who were working in the expo collecting and entering data saw this work as a burden and imposition at the end of a long day.
3. Theories of Change can be illusive when new ground is being broken
- Although both CIRMA and IDRC had a general idea of the expo project`s theory of change, we did not spend enough time actually articulating our assumptions and mapping out the potential complexities and variables – psychological, social/class, gender, political – that would influence the reactions by individual visitors who viewed the museum expo and the subsequent formation of attitudes.
-In hindsight: Theory of change was far too simplistic. Using the expo to induce behavior change among individuals was viewed as an important proxy for assessing social change and was one of the motivating factors behind the selection of Outcome Mapping as a central methodology.
-A more rigorous, participatory and externally validated process for thinking through the theory of change underpinning the expo would have allowed us to view and understand the expo as a tool for cognitive reframing-- confronting individuals with information discrepant or contradictory to their expressed attitudes or self-image to create opportunities for re-framing and re-organisation of these attitudes
- As it was, we missed the mark, and in this case, behavior change was never an appropriate indicator of social change – mainly because the viewing of the expo was in itself a passive activity which would have needed to be accompanied by clear strategies for social action - if actual changes in behavior were to be the final outcome.
In order words – the expo could not work in isolation – changes in attitude are part of a wider, multi-causal package.
-Admittedly, we fell into the project trap: We were perhaps caught up in the short timeline of OUR project intervention, looking for larger impact pay-offs -- the sorts that involve multiple actors and are inter-generational -- We assumed a more linear process of knowledge transfer what would result in attitudinal change – ironically, the Outcome Mapping Framework was extracted out of the wider system -
--However, given the cutting edge nature of the national campaign and the fact that both CIRMA and IDRC were breaking new ground, it was only through the evaluation process that we were able to fully grasp the complexity of the theory of change underlying the expo and its use. We found deficiencies in the original intentional design and of course the implementation failures that I noted earlier on.
This error has now been rectified.

8. How the story ends… Re-launch of expo, February 2009
New pedagogy & strategy for teacher training
Part of history curricula
M & E strategy redesigned: Clearer intent, multiple, mixed methods
Challenges of integrating M & E into donor strategies
HOW THE STORY ENDS
With the re-launch of the expo in February 2009, the focus has been on the development of a pedagogic strategy both to help teachers to digest the issue and to become constructive participants in post-exposition dialogues and investigation in the classroom.
In using the expo in a more directive, targeted manner, the expectation is that teaching professionals in Guatemala can begin to move away from more traditional forms of emphasizing historic facts, not enquiry, and that students will begin understand and problematize why accounts of the same event might differ.
- With its re-design and re-launch in February of 2009, the expo is now being used as a tool for training primary and secondary teachers and public servants on issues of race relations.

9. Thank - you! Note: Photographs in this presentation were taken from Learning Workshop on Representing Class, Ethnicity, Gender, and Race to the Public, (Powerpoint presentation) December, Tani Marilena Adams, CIRMA, Guatemala