1. A Realist Evaluation of Performance Management for Social Services Lessons from the implementation of “Outcome Management” in the voluntary sector in Singapore Robyn Tan, PhD candidate Institute of Development Policy and Management University of Manchester Email: robyntan_ht@hotmail.com

2. “Public sector performance measurement is, in effect, like putting a meter on a black box: we have little knowledge of the mechanism inside and no theory linking inputs, processes, outputs and outcomes to explain why a particular result occurred or to prescribe what management or organizational adjustments are needed to improve performance” (Grenier 1997)

3. The core of performance management is for achieving accountability and improvement - Yes? No? Not sure?

4. Research Aim To examine how and why performance management works (or not) to achieve accountability and improvement among voluntary organisations, through the use of performance information

5. Case Example: Outcome Management for “Job Placement and Job Support” Outcome Statement Out of 55 clients who were engaged in job placement interview, 40 stay in the job for at least 6 months. Milestones Targets Measurement Tools 1.Clients are engaged in a job placement interview 55Registration Client Progress Report 2. Clients are engaged in the development of employment plans 55Programme record Evaluation Report 3. Clients are matched to suitable jobs and attend job interviews 50Placement Record 4. Clients accept job offer 50Placement Record 5. Clients stay in the job for at least 2 weeks 50Placement Record 6. Clients stay in the job for at least 3 months 42Feedback form Site visit Case notes 7. Clients stay in the job for at least 6 months 40Feedback form Site visit Case notes For practitioners to meet specified targets, in order to maintain accountability For practitioners, (1) To learn about client’s progress in relation to the milestones; (2) To improve the programme

6. Applying realist evaluation to the study of performance management Realist evaluationPrimary data sourcesSecondary data sources  Develop the initial programme theories of how and why Outcome Management is expected to work NILPolicy papers Training manual Outcome Management related literature Performance management literature  Test the actual theories of how and why Outcome Management works against the initial theories, using 7 identified case studies Individual & group interviews with Council programme administrators and voluntary organisation practitioners Social service contract Performance reports  Develop refined programme theories of how and why Outcome Management would work Data analysis from primary and secondary data sources

7. How to unpack a complex intervention such as performance management?

8. Implementation chain of Outcome Management Step 1: Identify initial, intermediate & long term outcomes Step 2: Specify programme in relation to outcomes Step 3: Select performance indicators & measurement tools Step 4: Set & meet targets Step 5: Use performance information for learning & improvement

9. How to identify the outcomes for the study? “Outcome Management is intended to serve as a robust and multifaceted oversight system for social services and to improve the overall service standards of service delivery” – National Council of Social Service

10. Developing initial progamme theories: Identifying sub-outcomes & higher-order outcomes Step 1: Identify initial, intermediate & long term outcomes Step 2: Specify programme in relation to outcomes Step 3: Select performance indicators & measurement tools Step 4: Set & meet targets Step 5: Use performance information for learning & improvement Oi: Voluntary organisations formulate initial, intermediate & desired outcomes Oii: Voluntary organisations specify prog activities or intervention strategies in relation to the outcomes Oiii: Performance information generates feedback on client progress in relation to the outcomes O1: Voluntary organisations meet targets, in turn achieve accountability to the Council O2: Voluntary organisations learn about clients’ progress O3: Voluntary organisations make actual changes to improve the programme

11. Initial programme theories & actual programme theories Outcome Management Initial programme theoriesActual programme theories Step 1: Developing outcomes C: Institutional requirement C: Equipped with necessary knowledge and skills M: “Acquiescence” Oi: To formulate well-defined outcomes C: Limited knowledge and skills C: Intervention-specific context M: “Mimicking” tactics M: “Compromise” tactics O-: Fail to formulate well-defined outcomes Step 2: Specifying programme C: Lack of institutional requirement C: Equipped with necessary knowledge and skills M: “Tight coupling” Oii: To specify programme in relation to the outcomes C: Lack of institutional requirement; depends on organisational discretion C: Intervention-specific context C: Limited knowledge and skills M: “Decoupling” O-: Fail to specify programme in relation to the outcomes Step 3: Adopting measurement tools C: Institutional requirement to measure outcomes quantitatively; choice of measurement tools based on organisational discretion C: Equipped with necessary knowledge and skills M: “Tight coupling” Oiii: To generate feedback on client progress in relation to the outcomes C: Choice of measurement tools based on organisational discretion C: Limited knowledge and skills M: “Decoupling” M: “Pragmatism” O: Lack of standardisation in the use of measurement tools O: Lack of routine monitoring O-: Fail to generate feedback updates on client progress Step 4: Setting and meeting targets C: Institutional requirements to meet targets M: “Target system” M: “Ranking system” M: “Signal function” O1: To meet targets in order to achieve accountability M: “Target system” O-: Targets were met BUT they were measured against poorly-defined outcomes Step 5: Using performance information Ci: Well-defined outcomes Cii: Specification of programme in relation to outcomes Cii: Generated feedback on client progress in relation to outcomes M: “Feedback” mechanism O2: To learn about clients’ progress in relation to outcomes O3: To make actual changes for improvement Ci-: Poorly-defined outcomes Cii-: Fail to specify programme in relation to outcomes Ciii-: Fail to generate feedback on clients’ progress O2-: Fail to learn about clients’ progress O3-: Fail to make actual changes for improvement

12. Key findings Of the 7 case study programmes, only 1 achieved accountability and improvement Lack of institutional requirement for voluntary organisations to articulate and test the underlying theory of change (institutional context) Outcome Management is applied to the measurement of outcomes of a wide array of social services without considering the programme- specific characteristics (intervention-specific context)

13. Key findings (II) Programmatic interventionSingle practitioner-driven intervention e.g. Job placement and job support; sheltered workshop e.g. Counselling; case management Reasonably standardised programme across practitioners Individualised interventions based on practitioner’s choice of theoretical approach and needs of clients One overarching theory of changeThere are as many theories of change as there are practitioners and clients Practitioners deliver programme according to the articulated theory of change Practitioner delivers intervention based on his or her own theory of change (whether articulated or not). Facilitates learning and improvement across practitioners Learning and improvement (if any) is based on practitioner discretion

14. Developing refined programme theories Step 2: Identify initial, intermediate & long term outcomes Step 3: Specify programme in relation to outcomes Step 4: Select performance indicators & measurement tools Step 5: Set & meet targets Step 6: Use performance information for learning & improvement Step 1: Articulate theory of change underlying programmatic intervention

15. Reflections Identifying the level of abstraction for data collection and analysis – higher or lower or at what level? Identify the level of abstraction that can provide relevant theoretical and practical insights required for the study

16. Reflections (II) Identifying mechanisms from contexts – which is which? And does it matter? e.g. “The Council gives voluntary organisations discretion over choice of measurement tools” (context) “Voluntary organisations exercising their discretion for their choice of measurement tools” (mechanism) Identify the context-mechanism dyad responsible for generating the outcome

17. Reflections (III) Making sense of the contexts for the development of refined programme theories Categorising contexts according to their layers of social reality – individual capabilities, organisational context, institutional context, intervention-specific context Identifying the “order” in which they are likely to work to give the intervention a reasonable chance of success.

18. Thank you