1. Increasing Evaluation Transparency: A Dialogue Strategy Sheila A. Arens June, 2003

2. Context: Educational Rhetoric Movement toward “evidence-based”  Accountability for public monies US: reauthorization of ESEA (Jan ’02) Canada, Europe But how transparent are evaluation findings? Or how well do we communicate findings to (or with) broader constituencies?

3. Evaluative Findings On Stage In research, positive results are more likely to be published in academic journals – positive bias In evaluation, findings are sometimes shelved or disregarded  Why? Propriety; dismissive of findings (fail use or validity tests), do not resonate with beleifs Other instances where evaluation is prematurely given “top fold” status

4. Paradox: Too much information? While public attention to educational research and evaluation findings over the past several decades, public faith in education and in its research has decreased (Heath, 1999) What is our responsibility for making our work transparent? What role ought evaluation serve?  Ethical / moral questions Should transparency be contingent and relative? Should transparency underlie all work?

5. Participatory Evaluation Capacity to increase transparency; openness / democracy But evaluators have differential understandings of “participatory”  Emancipatory; transformative; utilitarian  Depth vs. breadth of inclusion Considerations are moral / ethical

6. Strategy for Public Engagement Strategies pursued as data collection efforts or throughout the evaluation? Strategies for gathering community input: Surveys, polls Interviews Focus groups  Engaged /deliberative dialogue

7. Deliberative Approaches: Rationale Wanted approach that constructively engaged & was inclusive (reflected democratic commitments) Reasoned conversations/ deliberative dialogue: approach that enables positions & contrasting ideologies to surface and be co-explored Encourages participants to consider collective & personal roles w/ respect to social problems & solutions

8. Example: Educational Standards Through engaged dialogue, we sought to uncover: The public’s perception of standards- based education, and The extent to which the public would support low performing schools

9. Methods Conversations with groups of citizens convened Participants represented wide array of stakeholder groups Used conversation framework to initiate Field notes, video tapes and observations collected

10. Data Analyses Analysis: notes & videos  analytic inductive process: separately reviewed, formulated tentative assertions based on emerging themes Compared assertions & sought disconfirming evidence

11. Findings Initially common perceptions (large-scale survey findings) reflected However, conversations led to participants more carefully examining their own assumptions about what is meant by:  Standards  Assessment  Accountability

12. Support standards, assessment, accountability Accountability measures: Reliable? Valid? Current accountability fails to address concerns Acknowledged myriad (external) factors Saliency of non-academic issues Support for LPS [in principle] (con’t)

13. Engagement Toward Understanding Large-scale evaluations grounded in surveys:  Fail to reveal nuances of perceptions  but also fail to engage participants toward increased understanding of own, others’ and commonly held ideological positions / stances emerged Public awareness increased along with ability to understand and engage in conversations

14. Next Steps?? Initial dialogues can help shape evaluative outcomes (in present case, relative to educational standards & accountability) Add’l research: extent to which participants…  Are more informed & engaged  Are better positioned: policy discussions & decisions  Better understand: evaluation processes, influence evaluation questions, understand evaluation outcomes