Everyone’s Story Counts – an overview of issues regarding measuring social impact in the community sector - Barbara Bloch, CCS UTS “Not everything that counts can be counted. And not everything that can be counted, counts” – Albert Einstein - A Einsten

Definitions of Social Impact  Productivity Commission 2010 ‘four levels of impact mapping’ – they are the measurement of inputs, outputs, outcomes and impacts. Impacts are distinguished from outcomes in that they are broader, they try to capture long-term effects on individuals and the community (PC:34).  Lyons and Zappalà (2009) Micro level: impact of individual programs (aka evaluation) Meso level: impact of individual organisations Macro level: ‘identifying and measuring the contribution or impact of the social economy or the slightly smaller nonprofit sector’ (3)  EGO Consultancy Melbourne (2010) Measuring and communicating social impact is emerging as a key skill-set in today’s social economy. The systematic capture and analysis of the impact you make is a crucial means to improving the performance of your project, initiative or venture. Importantly, research and measuring social impact are not just for academics. Everyone can do it, providing you’re down with the know-how and are equipped with the right set of tools. [ 17 October 2011]

Organisations interviewed  NCOSS  Information Cultural Exchange Parramatta (ICE)  Local Community Services Association (LCSA)  Western Sydney Community Development Forum  South-West Sydney Community Development Organisation  Metro Migrant Resource Centre, Campsie  Parramatta Community Pals  Metro Lands Council  Canterbury City Community Centre, Lakemba  Illawarra Youth Services  Ethnic Communities Council (ECC)  Bankstown Area Multicultural Network (BAMN)

The difference between social impact assessment and program evaluation Program Evaluation (PE)  Most commonly used, and most concerned with reporting outputs, due to pressure from funding bodies  Tends not to ask or answer questions like does the program / service solve a problem? Are people being helped? What are the outcomes and for whom?  Interviewee: ‘Measurement exists only if you can touch it, see it and count it’ Social Impact (SI)  Much broader than PE  Long term measurement  Looks beyond the stated objectives of a particular program or service  Trying to capture unintended consequences  About client and community outcomes, and it asks the question: What is making a difference?

How do you measure your organisation’s social impact?  Noone in my survey used any of the common SI frameworks, eg Social Accounting Audits, Logic Models or SROI – Social Return on Investment  Examples of practices a few organisations used: - Form of triangulation of evidence through collecting different types of data eg client interviews, across agency, feedback from external partners - note tracking client impact doesn’t necessarily become social impact - Count where and what you can, gather qualitative feedback from different sources, track web and social media interactions and collect stories to give a richer context, feedback from external stakeholders.  Results based assessment (RBA) – significantly distinguishes between performance and population accountability

What issues do you face in trying to measure social impact? Indeed we need measures around social impacts - in my experience it's what do we use as indicators, how do we make time for it and commit resources to it and what type of research methods are useful with our target groups? ( comment to researcher)  Attribution Given the length of time that needs to lapse before being able to measure SI, and the fact that no organisation works in isolation, how can even the best indicators attribute changes in a population/community to one organisation, with certainty?  Linear Causality How can you be sure that x caused y…..people can be influenced by other factors apart from direct service provision.  Time and data management Reporting mechanisms are not designed to assist organisations with planning, nor was there the capacity to do analysis over time of programs and services. Immense frustration about data collection - the limited resources for organisations to use the data they collect, to be able to identify trends, especially significant in relation to impact, with its need to look longitudinally.

Collective Impact  Collective impact, ‘is the commitment of a group of important actors from different sectors to a common agenda for solving a specific problem… It differs from networks, partnerships, collaborations partly through its scale (large), through having a centralized infrastructure, a dedicated staff and a structured process that leads to a common agenda, shared measurement, continuous communication and mutually reinforcing activities among all participants’ (Kania and Kramer 2011: pp 36, 38).  Five Conditions of Collective Success o Common Agenda o Shared Measurement Systems o Mutually reinforcing Activities o Continuous Communication o Backbone Support Organisations

Some concluding questions  What about the bigger picture of social change?  Whose agendas? Whose measures?  Will there ever be a tool to capture everything we do?

References  Kania, J and Kramer, M (2011), ‘Collective Impact’, Stanford Social Innovation Review, Winter.  Zappalà, G and Lyons, M (2009) Recent approaches to measuring social impact in the Third sector: An overview, CSI Background Paper No 5, July.  Keevers, L, Treleaven, L, Backhouse, H and Darcy, M (2010), Practising Social Justice: Measuring what matters, Illawarra Forum Inc: Jamberoo NSW.  Lee, P (date unknown), What’s Wrong with Logic Models? LCSA: Occasional Paper No. 1.  LCSA (2008), ‘Neighbourhood Centres and Results Accountability. A Conversation with Mark Friedman’, Local, The Newsletter for Community Development in NSW, LCSA.  Productivity Commission (2010), Contribution of the Not-for-Profit Sector, Research Report, Canberra.