Evaluation on shifting ground Sonia Barnes, NZ Police Alison Chetwin, Independent Evaluator (formerly NZ Police)

Presentation overview Context Neighbourhood Policing – evaluation approach and changes to it, specific learning and contributions Mobility – evaluation approach and changes to it, specific learning and contributions What we learnt

Context Policing Excellence Suite of 9 initiatives aiming to improve efficiency and effectiveness of NZ Police and shift more effort into prevention Implementation of change programme began in 2010 with formal project management processes and evaluation Implementation completed in 2013, targets to be met by 2014/15

Context 2014/15: achievement of Policing Excellence aims –Increase preventative policing by 4% –Reduce crime and crash by 13% –Reduce prosecutions by 19% 2017: contribute to Better Public Services targets –Reduce crime by 15% –Reduce violent crime rate by 20% –Reduce youth crime by 5%

Neighbourhood Policing Neighbourhood Policing Teams: – Sergeant – Vulnerable neighourhoods – Up to 9,000 residents –Community engagement, partnership working, problem solving Aim to: –Reduce crime and crash and re-victimisation –Increase trust and confidence, satisfaction, feeling safe

Original evaluation approach South Auckland – to inform national roll-out Desire for outcome evaluation –Two years before measure –Crime reduction –Perceptions of Police Formative evaluation – Identification of outcomes and theory of action – Activities based on good practice – Early operation of teams

Adapting the evaluation to implementation changes Accelerated implementation –Implementation in South Auckland late 2010 – early 2011 –Evaluation commenced early 2011 –Directive mid-2011 for 32 NPTs by end 2011 Adaptation of evaluation approach –Emphasis on formative evaluation and learning to inform further implementation –Advisory role on implementation team –Responsive to ad hoc evaluative requests

What the evaluation contributed Wider awareness and utility of findings Evidence based development of initiative: –International research evidence –Learning from local implementation Implementation team sought advice from evaluation team on aspects of development Education/ awareness raising of utility of evaluative work – raised profile of value of evaluation

Mobility initiative Mobile devices to access and record into police systems while out of station Aim to save time, improve safety, make police work more effective Trial in four police areas to be evaluated Emphasis on summative evaluation

Mobility trial Model of evaluator/implementer collaboration –Early involvement as independent expert adviser –Input into trial design to ensure evaluability –Reporting to allow time for outcomes to emerge –Integration between project management and evaluation –Formal agreement to evaluation framework –Investment decision to draw on evaluation evidence

Mobility – outcomes framework

Shifting timeline April 2012 Trial commences Jan 2013 Summative evaluation report Feb 2013 Business case for national roll-out June 2013 National roll- out commences Oct 2012 Business case for national roll- out Jan 2013 National roll- out commences

Options – changed timeframe Continue with agreed evaluation plan Seek reversion to agreed timeframe Abandon evaluation Adapt evaluation to new needs

Adapted design Need to refine the trial –Formative real time information – surveys at completion of each roll-out –Informal verbal reports – surveys and observations Need to inform national roll-out –Strengthen and bring forward qualitative component –Formal interim report and presentation Need to inform investment decision –Retain summative evaluation

What we learnt Importance and value of working with project management methodologies Flexibility expands evaluation uses –Being prepared to circulate emerging findings leads to greater utility –Being willing to assist in developing the initiative results in initiative being more grounded in evidence Summative evaluation needed for large investment decisions Accelerated timeframes not always achievable Demonstrate value of formative evaluation

Contact details Sonia Barnes Senior Evaluator, NZ Police Alison Chetwin Independent Researcher & Evaluator clear.net.nz evaluation-reports