1. Evaluating ‘Social’ & Health Projects Presentation by Elliot Stern International Seminar on Professional Education & Health Salvador, Bahia July 2006

2. Evaluating Social & Health Projects Address three topics How evaluations in ‘social’ arenas such as vocational training in health are designed and implemented How we ensure evaluations are used or institutionalised What evaluation questions should be asked in evaluations of professionalisation and social/healthcare

3. Evaluating Social & Health Projects Evaluation strategies build on two foundations: The rationale, purpose and types of evaluation The characteristics of what is being evaluated

4. Evaluating Social & Health Projects Evaluation purposes include: Better programme planning and design Improving programme management & delivery Demonstrating success in terms of results Explaining and predicting what works Strengthening social networks and institutions Some evaluations are about learning and some about accountability – much of which is about timing and programme stage

5. Evaluating Social & Health Projects Main types of evaluation - Allocative / Economic Management / Performance Causal / Explanatory Formative/Developmental Participatory

6. Evaluating Social & Health Projects Underlying these types of purposes there are of course value choices and priorities According to whose priorities are evaluations conducted? Who benefits and can use the results? There is always the question of who are the stakeholders and legitimacy of different groups and interests

7. Evaluating Social & Health Projects Matters of values are essentially political, moral and ethical questions, but it also depends on what we are evaluating Social, health and pedagogic and training programmes are not the same as social security, transport or information technology programmes

8. Evaluating Social & Health Projects ‘Social’ has many implications for evaluation – ethical and pragmatic Social emphasises that the ultimate beneficiary of programmes are citizens and communities as well as officials and professional groups Social highlights that the perspectives of service users or beneficiaries of development programmes have knowledge and rights It makes it likely that programmes will be better designed, targeted and implemented if the voices of ‘social’ stakeholders are included Social policies and programmes are often multi- dimensional requiring a combination of interventions

9. Evaluating Social & Health Projects This has implications for evaluation approaches and methods Need to use participative and consultative methods Expect ‘social’ stakeholders to have a voice in programme design stages as well as to provide ‘user’ judgements The need to monitor social aspects of the programme implementation process The need to use multiple methods and synthesise data from many sources

10. Evaluating Social & Health Projects Yet the term social in the world of social programming and policy making has still wider meanings. It encompasses for example service provision and social development And the problem with social development is that it cannot be predicted, it is not static – it develops, which also leads to problems for evaluation

11. Evaluating Social & Health Projects Evaluators who will at some point be asked to judge ‘success’, need to criteria to make their judgements. These criteria are usually derived from programme objectives, but if these objectives change or are unstable……. Which is why specific approaches are needed….

12. Evaluating Social & Health Projects In social development programmes, process evaluations that reveal why objectives change or mutate are often more useful than objectives driven evaluations Evaluation can also help to build capacity by empowering participants in social programmes making it easier for them to participate, manage their role and clarify their priorities and strategies

13. Evaluating Social & Health Projects This introduces not only evaluation purpose but also questions the methods, style and role of the evaluator Is the evaluator an observer? Technician? Participant/protagonist? Referee? Advocate? In social programmes the technician/observer role is often difficult to maintain nor is it effective

14. Evaluating Social & Health Projects If evaluations are to be used and useful they need to be built in to the institutional arrangements that manage and coordinate programmes The growth of the (new) public management has seen a growth also in internal evaluation functions that are closely tied to programme and policy management

15. Evaluating Social & Health Projects Public management today is characterized by: Multi-level governance and adjustment to local as well as national conditions Interagency & public private partnerships Temporal/organisational instability Active engagement with civil society New consultative, dialogical and consensus oriented arrangements A pre-occupation with risks, efficiency and performance

16. Evaluating Social & Health Projects This creates demands for new skills and competencies from evaluators. Being able to: Translate evaluation data into policy advice Build generalised conclusions from very diverse implementation contexts Speak directly to different civil society groups Synthesise the experience of different stakeholders and ‘partners’

17. Evaluating Social & Health Projects So evaluations have to look in two directions: Inwards to government & public managers enabling them to manage and adjust during implementation and improve performance on a continuous basis Outwards to officials, decentralised administrations, partnerships and citizens so as to integrate their experience and maintain a dialogue and consensus

18. Evaluating Social & Health Projects I have spoken of some of the characteristics of social programmes. One of these is that they are multi-dimensional, using many policy instruments in combination. This has implications for where we get our evaluation questions from. I distinguish here between evaluation and monitoring. Evaluation tries to answer evaluation questions – and these are usually derived from formal bodies of knowledge and from theory.

19. Evaluating Social & Health Projects Two examples of evaluation questions I have about PROFAE and its successors that I have not yet heard addressed at this conference One concerns professionalisation and the other the evolution of the service society

20. Evaluating Social & Health Projects Professionalisation implies: Self regulation rather than control by education or certification; the market; managers; or by government regulation Authority defined by ‘special’ knowledge and a willingness of consumers to accept this authority Exclusive rights to undertake certain work

21. Evaluating Social & Health Projects Professionalisation’ strategies for occupational control exist: ‘where the tensions inherent in the producer- consumer relationship are controlled by means of an institutional framework based upon occupational authority. This form of control occurs only where certain conditions exist ……’ (Johnson 1972) These conditions include a large market of diverse consumers with ‘diverse’ interests, who are not organised and are largely dependent. Where there is ‘a single client or a small group of powerful clients’ professional strategies of control are less likely to be successful.

22. Evaluating Social & Health Projects Irrespective of whether the objective conditions for professional strategies for occupational control exist, the evaluation question remain: What are the consequences of increasing the status, authority and capacity for self-regulation of any occupational group? What are the risks of rigidities, demarcations and boundary disputes occurring between newly strengthened professionalized groups?

23. Evaluating Social & Health Projects A second example of evaluation questions for this and future evaluations of PROFAE type programmes concerns the evolution of service economies and risks that this evolution sometimes undermines autonomy and self-regulation among citizens and in communities

24. Evaluating Social & Health Projects Health innovation in Brazil attempt to combine two tendencies: Greater competence, training and skill among ‘professionals’, and Continued democratic, participatory and inclusive involvement of civil society Good example the role of community agents in Atenção Básica alongside nurses with ever-greater skills

25. Evaluating Social & Health Projects So the evaluation questions that need to be asked are: What are the consequences of these two tendencies – professionalisation and participation? Is it likely that the long-term consequences of ‘professionalisation’ will be to undermine autonomy and self regulation within communities and introduce a ‘dependency’ culture that will be costly and socially divisive? and, If so how can this be avoided or reconciled?

26. Evaluating Social & Health Projects In the last two examples I have been deliberately provocative. This is to illustrate the importance of theory as well as methods in evaluation. That we must not only use appropriate methods to gather information but that we must also ask the right questions

27. Evaluating Social & Health Projects In a programme and policy arena as complex as PROFAE and its successors this requires that we draw on a wide range of relevant theories: in public management, pedagogy, public health, welfare and dependency theory, labour markets, professionalisation and social development to name but a few!

28. Evaluating Social & Health Projects If evaluation is to become more than monitoring, contribute to programme management and social development it must: Use diverse theory & methods matching the complexity of what is being evaluated Be inclusive and open to stakeholder voices from across civil society as well as the public sector Relate to processes as well as outcomes. Address the needs of policy makers and public managers whilst not undermining social inclusion and the voice of civil society

29. Evaluating Social & Health Projects Good evaluation is not easy! But innovative and exciting programmes of the kind that exist in the social and health sectors in Brazil deserve equally sophisticated evaluation approaches.

30. Evaluating Social & Health Projects