Hosted by: Funded by: Tensions of Measurement: community investment activities in housing associations David Mullins and Vanessa Wilkes, TSRC Housing Studies Association Conference 15 th April 2011

Summary Sector Context Evidence Base Levels of Measurement – 5 levels of measurement – Questions and Issues Tensions – External Drivers – Internal Drivers Conclusions Further Research

Sector Context Well established industry, large enterprises (50% market share in social housing) Entrenched hybridity (social & commercial) Core business highly regulated & measured Non-core ‘community investment’ (CI) activities symbol of independence Legitimacy & control issues brought to fore – 2008 Housing & Regeneration Bill & NHF Audit New tools emerging to measure social impact of CI activities (in-house & off the shelf)

Evidence Base Scoping study – emerging approaches to measurement of social performance (2010) – literature review & checklist (17 tools to set priorities, measure outcomes or link these to corporate goals) – 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced) – 2 workshops (now 3, plus follow up tracking e.g. of first sector specific tool) NHF Neighbourhood Audits 2008, 2011 – Sector overview of CI (64% coverage by stock) – Inputs/Outputs (£435mil CI in 6,800 projects 2008; leverage 1.6:2.7) Linked PhD – Impacts & outcomes, depth case studies – More on this later!

The levels of impact measurement OECD definition of impact measurement “the impact of all changes, positive or negative, attributable to the intervention” Also – intended and unintended effects and long-term as well as short- term Original argument (Zappala and Lyons 2009): 3 levels of impact measurement – Whole sector – Individual organisations – Individual projects Tools, frameworks and methodologies not easily transferred between levels Framework developed in relation to housing study (Mullins 2010) to 5 levels

5 levels of impact measurement 1.Sector level 2.Organisational level 3.Team level 4.Project level 5.Individual Transferability between levels is dependent on: How the organisation adopts and uses the tool Having the necessary skills and expertise Relationship between internal and external data needs A corporate awareness of what is required from impact measurement The understanding and support by staff Maximising the usage of the data at more than one level

Questions and Issues Whole sector – Is it possible or feasible to measure impact? Dependent on collective approach and understanding Expensive Housing sector – disparate projects and circumstances – Draw on information from level 2 Organisation / Team / Project – Reason / motivation External requirement? Internal legitimisation / accountability? A learning tool? – Availability of skills and expertise Not necessarily new methodologies but alternative application Interpretative and subjective decisions – Draw on information from levels below

External Tensions External Drivers Funding Sector Promotion Dances around regulation Common outcomes / community budgeting Wider third sector policy influences (e.g. promotion of SROI) Emerging Tensions Data for Funders – Conflicts with internal needs – Multiple monitoring – Timescales – Mission Drift Sector promotion – ‘more than a good story’ – Limited no of PIs obscure complexities of context & impact

Internal Tensions Internal Drivers Accountability & Reporting (variety of audiences) – Boards & Trustees – Management – Making sure people enjoy the projects Benchmarking Organisational Power Play – Promoting visibility of CI within corporate scorecards – Aligning CI activities to corporate goals Emerging Tensions Commercial & Social Goals Social Dividend from commercial success – Tendency toward harder measures – leverage Contribute to core business – Attribution problems – leads to focus of CI on tenants Corporate v Project Levels – Who sets the goals? – which CI outcomes get reported?

Conclusions Early days but likely to grow High Expectations in advance of evidence! Tensions around types of measures and uses made of the data (both external & internal) Is it worth it? (practical issues, costs motivation/training) Underlying tensions reflect hybrid and large scale of these organisations & competing interests (e.g. Core business v CI)

Further Research 1.The role of community investment measurement tools within the organisation – How is community investment integrated and understood? – What measurement tools are used, what is their strategic fit within the organisation? 2.Data considerations (internal and external) – How is data reported and used? – What level of understanding does it provide? 3.The practical implications of measurement – Does impact measurement cause organisations to rethink their mission, goals or values? – What are the operational issues and practical considerations which affect measurement, such as length of contracts

Thanks for Listening Please Keep in Touch David Mullins Vanessa Wilkes