Developing a Monitoring and Evaluation Framework: Insight into internal stakeholder learnings Beth Ferguson AES Conference Sydney 2 September 2011

Outline Background and context Monitoring and evaluation framework and tools Implementation of the framework Stakeholder experiences and learnings

Background Immigration New Zealand funds a variety of services to help new migrants and refugees to New Zealand to settle in to their new home $5m of funding towards 30 contracts 18 services are Settlement Support New Zealand (SSNZ) initiatives 3 skills matching programmes

Background In 2008, a monitoring and evaluation of settlement services was initiated. Part of a government commitment to: ‘strengthen settlement services by establishing a robust evaluation process to ensure effectiveness’

Purpose Purpose was to develop a strategy for a monitoring and evaluation framework to: determine the overall quality or value of services the Department funds contribute to the process of continuous improvement for existing services Ongoing monitoring and evaluation rather than a one-off evaluation

Rationale Developed to monitor immediate outcomes of settlement services funded by the Department of Labour to: assist with self-learning and self-assessment make criteria explicit and information needs transparent be integral to the business and contract management enable immediate outcomes data to be collected, and ensure sustainability over time and minimal requirements for service providers.

Stakeholders Implementation & Development: Migration Research For use by: Settlement Purchasing Team at the Department of Labour Providers of settlement services funded by the Department

The Framework Tools Good Practice Matrix Service logics Provider feedback Surveys Mid-year & Full year reports

Good Practice Matrix Based on an international literature review of good practice Will be modified/updated as new evidence becomes available Helps services to identify areas of strength and areas to potentially focus on

Assesses alignment with Good Practice Principles: Infrastructure Targeting Migrant Involvement Integration of services Service Delivery Across 3 levels: Preferred practice Good practice Acceptable practice Good Practice Matrix cont…

Service logics Used as a way to visually and logically clarify the links between inputs, activities, outputs and outcomes of the contracted settlement services. Logics will be used as a tool when contracting new settlement services Tool for services to map progress towards Key Performance Measures

Provider Feedback A tool for contracted settlement service providers to reflect on the work of their service and the delivery of the service against the contract. Service providers report on: what is working well for their service What is not working well Improvements that need to be made

Surveys Client Feedback Survey Designed to monitor trends and client outcomes Based on the Common Measurement Tool (CMT) Translated into 8 languages (Chinese, Korean, Spanish, Russian, Hindi, Tagalog, Japanese, Arabic)

Surveys Local Settlement Network (LSN) Survey For use by Settlement Support New Zealand (SSNZ) initiatives throughout the country Designed to assess how well SSNZ initiatives are working with local agencies to increase their responsiveness to the needs of newcomers

Reports Reports have been developed to summarise information from the monitoring and evaluation tools and to provide overall benchmarks Mid-year reports: Produced in February to update on progress Full-year reports: Comprehensive report produced in August Include all monitoring and evaluation tools

Implementation Trialled with 9 settlement service providers in May 2010 Some additions and refinements were made as a result of the trial Implemented November 2010 with 27 settlement services

Stakeholder experiences: Researchers/Evaluators Challenging process Valuable to up-skill/train others to use the tools in a consistent way Utilising surveys as an educative tool has been useful – has provided an opportunity to inform clients about the purpose of the service.

Stakeholder experiences: Settlement Purchasing team Practicalities translating a framework developed using the literature into an operational tool Need for ongoing support from the developers of the Framework need for ongoing funding to support the framework.

Stakeholder experiences: Settlement service providers Feedback predominantly positive Appreciative of transparency Keen to know how they can improve Some questions about the implementation Some concerns about use of results Elements of the surveys in particular can be labour intensive

Joint Learnings Need a common understanding from the beginning about: The problem and assumptions What is to be achieved from the framework. Need consultation and involvement in the development of the Framework by all stakeholders Need to discuss and agree on the viability of different approaches

Joint learnings Engagement with service providers Trial and pilot tools Training and support to those using the tools Use a tried and tested survey (CMT) Translations – complex and time consuming

Conclusions The Department has successfully developed and implemented an ongoing monitoring and evaluation framework However ….. Developing and implementing a monitoring and evaluation framework is complex: Variety of stakeholders Differing backgrounds Differing understandings of evaluation and purpose

For more information…. Migration research website: