1. AWARE Evaluation Results and Conclusions
Ronald J Pohoryles ICCR Foundation
Ensuring Knowledge Based Citizens' Participation
AWARE Evaluation Workshop, 20 October 2011

2. Overview Evaluation Methodology Results – Process Process and Outcomes
Conclusions

3. Accompaying evalution of the process

4. Evaluation Methodology
Purpose: To assess...
... the soundness of citizens' participation process
... its effectiveness in terms of outcomes
Evaluation Instruments:
Interviews
Observation
Paper questionnaires (for the citizens participating at the workshops)
Internet questionnaires (of wider groups of experts)

5. Results – Strength
On the whole, the assessment of the process has proven that the process was
well planned,
well-structured,
and largely carried out in line with the original plans formulated in the project design.  

6. Strength of AWARE
This was the case particularly for the local level activities.
Most organisational and procedural aspects were evaluated positively by both the participants/citizens and the team of observers.

7. Strength of AWARE
This includes the setting or framework in which the presentations and deliberations took place as well as the proceedings themselves. The latter is especially relevant since it covers the vital issues of how the discussions were structured, how individual views were heard, and how the opinions and facts of experts and citizens were finally integrated.

8. Three major challenges
Despite the overall positive assessment of the local proceedings, two distinct weaknesses need to be highlighted.
The first one concerns the complexity of the problems at hand lasting on the interaction between the citizens and the scientific experts.
The second challenge was the difficulty to involve policy makers on the local level.
The third challenge is the language issue.

9. Challenge I: The complexity of the issue
All experts were asked to make their presentations as understandable for a lay audience as possible.
With a few exceptions the presentations have proven successful.

10. Challenge I: The complexity of the issue
The experts were able to convey the full information available due to the short time allocated to them for their presentations.
In some cases incomplete information has led citizens to wrong assumptions that surfaced during the deliberations.

11. Challenge I: The complexity of the issue
This is due to the openness of the citizens' deliberations. It is impossible to predict which path the citizen deliberations will take and it is not desirable to determine this path beforehand.
This shows the necessity to involve all experts during the whole process in form of a 'knowledge repertoire'.

12. Challenge II: Low involvment of local policy makers
The second challenge is the difficulty to involve local policy makers.
Even when it was possible to involve policy makers it rather remained on political statements than opening up to an exchange of ideas with citizens and/or experts.

13. Challenge II: Low involvment of local policy makers
This is by no means an organisational weakness of this specific project. Experience shows that science and research are rather concerned with technical and pragmatic issues, policy making is based representing and/or weighting different interests.
The active involvement of the policy community in the process would more or less automatically lead in the direction of political deliberation, which in turn raises the issue of legitimacy of a random group of ten citizens.

14. Challenge III: The language issue
Specifically for citizens' participation at the European level, the language issue should not be underestimated.
The North Sea and Baltic Sea case studies were transnational cases, whilst the third one on a local level in Italy.
The integration of the transnational groups on the European level did not cause problems, whilst due to the language problems the integration of the Italian group was more difficult.
This is certainly an important lesson for similar exercises in the future. Where international/cross border problems need to be addressed all citizens need to speak a common language, which poses a challenge to the representativity of the group.

15. Process and Outcomes
Lesssons learned from the AWARE Project

16. Results - Outcomes
According to the results of the local surveys after the events, stakeholders and citizens seem to hold different perceptions.
Stakeholders generally state to have a good level of knowledge about the topics addressed in the workshop, while
citizens state to have a lower level of knowledge even though they had prepared themselves prior to the workshop.
Both stakeholders and citizens are aware that the presence of all actors is a fundamental requisite to make any change to the present situation.

17. Size of the groups and quality of deliberations
Due to the comparatively small number of citizens in each group and the professional organisation of the deliberations, a reasonable balance of attention was given to the ideas and opinions of each participant.
This was true not only for the smaller case study workshops, but also for the two larger European workshops.
Citizens who were not as outspoken as others could voice their opinions, which is proven by the observation protocols as well as by the assessment of the participants themselves. It is also positively reflected in the satisfaction of the citizens with the outcome of the process.

18. Differences between the local and the European deliberations
On the local/regional level, process and outcomes of the deliberations were seen as successful both by the citizens and the experts. Each of the three policy documents produced by the citizens were seen as a positive input into the political process.
Even though there are differences in the assessment of the three local declarations, the citizens were satisfied with how their views were heard, processed and eventually integrated into the final policy documents.

19. Differences between the local and the European deliberations
The process of developing the European declaration based upon the three regional/local declarations was generally seen more critical.
They citizens did not feel equally strongly their ideas and spirit of their local declaration reflected in the European document.

20. The experts' views
The experts acknowledged that the declarations brought a somewhat novel perspective to the problems,
but in many cases they assessed the declaration as being naïve and lacking practical experience rather than a constructive way forward.

21. Confronting experts and citizens
This highlights the fact that in all the participatory steps knowledge has been provided by all the participants, but in different formats and in different amounts
Expert knowledge was given mainly by scientists, tacit and local knowledge mainly by stakeholders and local policy makers.

22. The Role of the Citizens
During the entire process, the knowledge of the citizens about the ecological problems improved, and their awareness and commitment increased.
The declarations then can be seen as the closest objective measure of their knowledge and commitment.
However, the declarations are also constrained by the type and amount of knowledge that has been exchanged, and indeed reflects to a large extent the knowledge already held by the actors participating in AWARE.

23. Conclusions
The constraints are due to the nature of participatory activity.
Information that are presented to decision-makers are necessarily short
Hence some information was necessarily lost in the process.

24. Conclusions
Overall, there was a spirit of mutual respect between the citizens, the stakeholders and the experts.
Based on the discussions with stakeholders the participants agreed on the importance of concrete actions.
However, there was also scepticism voiced on the fruitfulness of the deliberations: Will further actions really take place?