1. Evaluation capacity-building: An insider’s tale of value- adding Diana Beere

2. Overview What is ‘evaluation capacity-building’ (ECB)? What is ‘evaluation capacity-building’ (ECB)? Why should internal evaluators undertake ECB? Why should internal evaluators undertake ECB? How can they feasibly do so? How can they feasibly do so? A case study A case study

3. What is ECB? Typically understood in terms of skills and knowledge development… Typically understood in terms of skills and knowledge development…  with the aim of doing more/better evaluations  targeted at staff who do evaluations

4. But ECB must also involve… ensuring use and maintenance of evaluation skills ensuring use and maintenance of evaluation skills creating and sustaining a market for evaluation skills creating and sustaining a market for evaluation skills ensuring conduct and use of evaluation becomes routine organisational practice ensuring conduct and use of evaluation becomes routine organisational practice developing and maintaining an evaluation culture developing and maintaining an evaluation culture

5. A definition ECB is the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine. ECB is the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine. Compton et al 2002

6. ECB must be directed at the whole program/organisation, not just staff who undertake evaluations. ECB must be directed at the whole program/organisation, not just staff who undertake evaluations. ECB is intentional and collaborative: purposive, attentive, and reflective work with others (Baizerman et al 2002). ECB is intentional and collaborative: purposive, attentive, and reflective work with others (Baizerman et al 2002). This suggests that…

7. Evaluators & ECB ECB and program evaluation are different, albeit complementary. ECB and program evaluation are different, albeit complementary. ECB is not usually part of the everyday work of evaluators. ECB is not usually part of the everyday work of evaluators. ECB is not the responsibility of evaluators alone. ECB is not the responsibility of evaluators alone. But evaluators can have a key role. But evaluators can have a key role.

8. Internal evaluators & ECB Internal evaluators can and should make ECB part of their everyday work. Internal evaluators can and should make ECB part of their everyday work. (At least) four reasons … (At least) four reasons …

9. 4 reasons… Self-interest! Self-interest! Commitment to evaluation (AES aims to improve the theory, practice & use of evaluation) Commitment to evaluation (AES aims to improve the theory, practice & use of evaluation) Best placed to engage in ECB Best placed to engage in ECB Responsibility to add value for their organisations Responsibility to add value for their organisations

10. Is it feasible in practice? Extra work? Staff, skills and resources often limited Extra work? Staff, skills and resources often limited But can be undertaken simultaneously with program evaluation But can be undertaken simultaneously with program evaluation Need not require extra resources Need not require extra resources Does require ‘dual vision’ Does require ‘dual vision’

11. ECB & value-adding ECB practices add value for organisations. ECB practices add value for organisations. Internal evaluators can integrate program evaluation and ECB practices by consistently focusing on adding value while also discerning value. Internal evaluators can integrate program evaluation and ECB practices by consistently focusing on adding value while also discerning value.

12. Weiss (1999): Utility is the primary purpose of evaluation: it is the name of the game. Utility is the primary purpose of evaluation: it is the name of the game. However, evaluators must have realistic expectations about how their work will be utilised and what it can achieve. However, evaluators must have realistic expectations about how their work will be utilised and what it can achieve.

13. (obviously!) Be realistic. (obviously!) Be realistic. But look for realistic / feasible ways to achieve more... to provide more value and maximise utility. But look for realistic / feasible ways to achieve more... to provide more value and maximise utility. Every evaluation presents opportunities to add value. Every evaluation presents opportunities to add value. This means …

14. A case study The programs The programs The challenges The challenges ECB features of the evaluation ECB features of the evaluation

15. The programs QPS Shopfront and Neighbourhood Police Beat Programs QPS Shopfront and Neighbourhood Police Beat Programs Politically and publicly popular Politically and publicly popular Unlikely to be discontinued Unlikely to be discontinued But concerns about effectiveness But concerns about effectiveness

16. The challenges Outcomes evaluations expected (assumed), but no value and insufficient data Outcomes evaluations expected (assumed), but no value and insufficient data Limited resources Limited resources Address clients’ concerns about program effectiveness (add value) Address clients’ concerns about program effectiveness (add value) Maximise utility (add value) Maximise utility (add value)

17. ECB features (1) Close collaboration with program managers - education about evaluation issues, possibilities Close collaboration with program managers - education about evaluation issues, possibilities Attention to needs of beat officers as key stakeholders Attention to needs of beat officers as key stakeholders Reports presented as guides to best practice Reports presented as guides to best practice

18. ECB features (2) Evaluation emphasised as key to best practice Evaluation emphasised as key to best practice Attention to organisational and cultural context (problem-oriented & parnership policing & SARA model) Attention to organisational and cultural context (problem-oriented & parnership policing & SARA model) Development of beat officers’ course Development of beat officers’ course

19. ECB features (3) Evaluation training included in beat officers’ course Evaluation training included in beat officers’ course Development of evaluation planning worksheet (dual purpose) Development of evaluation planning worksheet (dual purpose) Continued collaborative relationship with program managers Continued collaborative relationship with program managers

20. Points to consider… No ‘one size fits all’ approach to ECB No ‘one size fits all’ approach to ECB ECB is never complete ECB is never complete ECB work can/should itself be evaluated ECB work can/should itself be evaluated

21. Key messages Internal evaluators are well placed to participate actively in ECB and have a responsibility to do so. Internal evaluators are well placed to participate actively in ECB and have a responsibility to do so. The key to success is to integrate ECB with program evaluation - seeking always to add value while discerning / assessing value. The key to success is to integrate ECB with program evaluation - seeking always to add value while discerning / assessing value.

22. Any questions?

23. References Baizerman, M, Compton, D W & Stockdill, S H (2002). Summary and analysis of the case studies: Themes across the cases. In The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93 (pp. 101-107). San Francisco: Jossey-Bass. Baizerman, M, Compton, D W & Stockdill, S H (2002). Summary and analysis of the case studies: Themes across the cases. In The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93 (pp. 101-107). San Francisco: Jossey-Bass. Compton, D W, Baizerman, M & Stockdill, S H (Eds) (2002). The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93. San Francisco: Jossey-Bass. Compton, D W, Baizerman, M & Stockdill, S H (Eds) (2002). The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93. San Francisco: Jossey-Bass. Owen, J M (2003). Evaluation culture: A definition and analysis of its development within organisations. Evaluation Journal of Australasia, 3(1), 43-47. Owen, J M (2003). Evaluation culture: A definition and analysis of its development within organisations. Evaluation Journal of Australasia, 3(1), 43-47.

24. Owen J M & Rogers, P (1999). Program evaluation: Forms and approaches (2nd ed). London: Sage. Owen J M & Rogers, P (1999). Program evaluation: Forms and approaches (2nd ed). London: Sage. Stockdill, S H, Baizerman, M & Compton, D W (2002). Toward a definition of the ECB process: A conversation with the ECB literature. In The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93 (pp. 7-25). San Francisco: Jossey-Bass. Stockdill, S H, Baizerman, M & Compton, D W (2002). Toward a definition of the ECB process: A conversation with the ECB literature. In The art, craft, and science of evaluation capacity-building: New Directions for Evaluation No. 93 (pp. 7-25). San Francisco: Jossey-Bass. Weiss, C H (1999). The interface between evaluation and public policy. Evaluation, 5(4), 468- 486. Weiss, C H (1999). The interface between evaluation and public policy. Evaluation, 5(4), 468- 486.

25. Thank you