1. 1 Improving Accountability by Using Reflection and Self-Assessment Presentation to the Canadian Evaluation Society/Société Canadienne D’ Évaluation Vancouver, B.C. June 4, 2003 Session 302 Ivana Zuliani LAB at Brown University Research Associate Providence, RI Ivana_Zuliani@Brown.edu

2. Presentation by Ivana Zuliani, LAB at Brown University 2 What is the Strategy? Reflection & Self-Assessment

3. Source: Preskill, H. & Torres, R. Evaluative Inquiry for Learning in Organizations. (Sage, 1999). 3 What is the purpose of the strategy (in general)? Reflection: enables individuals and groups to review their ideas, understandings and experiences; and enables team members to explore each other’s values, beliefs, assumptions, and knowledge related to the issue of interest. Three types of reflection: Content reflection Process reflection Premise reflection

4. Source: REL Self-Assessment Report, 2001 4 What is the purpose of the strategy (in general)? Self-assessment is a strategy used to assess the quality of data, processes, and mechanisms in order to foster improvement and a culture of accountability.

5. Presentation by Ivana Zuliani, LAB at Brown University 5 How was the strategy used in the PI work? Self-assessment implemented on 3 levels: Individual laboratories Indicator subgroups REL system

6. Presentation by Ivana Zuliani, LAB at Brown University 6 Self-Assessment questions 1. What assumptions and interpretations were made of the data? 2. What challenges did we face? 3. What worked well? 4. What aspects could be improved next time? 5. What changes should/could be made to this report for 2003? 6. What do the indicator data for 2002 tell us about the REL program as a whole?

7. Presentation by Ivana Zuliani, LAB at Brown University 7 Strategy (reflection & self-assessment) characteristics: Developmental Collaborative Cyclical

8. Source: Fetterman, D., Kaftarian, S. Eds. Empowerment Evaluation: Knowledge & Tools for Self- Assessment & Accountability. (Sage, 1996). 8 Benefits “Taking stock” Documentation of work accomplished Openness allows for checks in personal biases and builds trust By internalizing and institutionalizing self-evaluation processes and practices, a dynamic and responsive approach to evaluation can be developed to accommodate shifts in the external environment.

9. Source: Chelimsky, E. & Shadish, W. Eds. Evaluation for the Evaluation for the 21st Century. (Sage, 1997). 9 Limitations Inherent bias Time Competency traps Absence of learning structures and forums

10. Presentation by Ivana Zuliani, LAB at Brown University 10 Lessons Learned Self-assessment and reflection complement the external data verification process.