1. Reporting and Accountability
Understanding Success
Anne Couillard
Settlement and Integration Policy Branch
IRCC
October – November 2017

2. Purpose
IRCC perspective on Settlement Program success and desired outcomes
Provide an overview of IRCC data collection, and the pros, cons, and gaps
Provide some highlights from data analysis
Present how this information is used and reported
Discuss next steps and ways forward 1

3. Outline Background What is Settlement Program success?
IRCC’s Settlement Program
Focus on Outcomes
What is Settlement Program success?
How do we know we succeeded?
What is IRCC collecting?
Examples of findings
Annual Project Performance Report (APPR)
Clients and services (iCARE)
How is IRCC using this information?
Pros, cons, and gaps in data collection
Considerations & next steps 2

4. Legislative Authorities
Background IRCC Settlement Program
Legislative Authorities
Canada’s Settlement Program fulfills obligations of the Immigration and Refugee Protection Act to:
“Promote the successful integration of permanent residents into Canada, while recognizing that integration involves mutual obligations for new immigrants and Canadian society.”
The Settlement Program aims to assists newcomers to overcome barriers to participation in social, cultural, civic and economic life in Canada and help to foster an integrated society
Services are funded through contribution agreements and delivered by third-party providers
Funding is allocated by province and territory and centrally managed national and pre-arrival funds. This does not include the transfer to Quebec
It delivers services to help newcomers acquire necessary language skills, ensuring their access to information, connecting them to community resources, building their knowledge of life in Canada, and enabling labour market participation
To reach these objectives, IRCC also ensures Sector capacity and professional development, standardization of best practices, and building knowledge of the Program’s clients
Integration is a shared role with involvement and engagement of multiple players, whose contributions and levers help to achieve longer-term outcomes and IRCC Settlement Program objectives 3

5. Background
Focus on Outcomes
There are a number of factors creating momentum for looking at outcomes, and for review, revision and refinement of data collection and reporting
Ministerial Mandate Letter: ‘…a rigorous approach to data in order to accurately measure outcomes.’
Government of Canada and Departmental emphasis on results and delivery, focus on outcomes
Settlement Program evaluation
Levels Plan - program modernization
Ministerial consultations with stakeholders

6. Intermediate Outcomes
Background
Program Design
Inputs and Delivery
Settlement Program
Terms and conditions
Funding
Logic Model with expected outcomes
Settlement Services
Based on priorities for services and programming and directives to manage contribution agreements
Needs Assessment
Information and Orientation
Language Assessment
Language Training
Community Connections
Indirect Services
Outputs
Immediate Outcomes
Intermediate Outcomes
Ultimate Outcomes 5

7. It starts with expected outcomes!
What is Settlement Program success? Settlement Program Expected Outcomes
Immediate outcomes
Consistent and responsive Settlement Program delivery
Access to IRCC- funded settlement services is facilitated
Increase understanding of client settlement needs and appropriate linkages to other services
Clients increase knowledge of life in Canada
Clients improve official language skills
Clients acquire knowledge, skills, and connections to prepare for the Canadian labour market
Clients increase participation in communities and social networks
Partners deliver responsive and coordinated settlement and community services
Intermediate outcomes
Clients access services that meet their needs
Clients make informed decisions about life in Canada
Clients use an official language to function in Canadian society
Clients participate in the Canadian labour market
Clients are connected to communities and institutions
Communities foster welcoming environment for immigrants
Ultimate outcomes
Canada benefits from the settlement and integration of clients
Outcomes for clients are a key focus but two-way street important
Specific level of outcomes for specific clients groups 6

8. How do we know we succeeded
How do we know we succeeded? Examples of performance measurement indicators
Measured by:
Outcomes
Performance Indicators
# and % of clients receiving information by service and topic
% of clients receiving information who indicated that their knowledge of life in Canada changed as a result of IRCC-funded services, by topic and overall
Immediate:
Clients increase knowledge of life in Canada
# and % of clients by employment status as indicated by client (working, have worked but not currently, looking for work)
Incidence of employment earnings for clients 1 through 5 years since receiving first service
Intermediate:
Clients participate in the Canadian labour market
Sense of belonging
Volunteering rates
Employment rates
Ultimate:
Canada benefits from the settlement and integration of clients 7

9. Pilots, experimentation
What is IRCC collecting? Measuring performance at Projects and Program Level
Narrative Reports/APPR
GCS
iCARE
Newcomer Outcome Survey
External Database Linkages
Outputs:
Number of clients and their characteristics
Number of services and their characteristics
Geographic distribution
Language progress
Inputs/Resources:
Budgets
Targets
Operational monitoring
Delivery:
Locations
Partnerships & resources
Challenges & opportunities
Successes
Needs assessment approaches
Performance assessment approaches
Ultimate Outcomes:
Economic, social and
cultural integration through iCARE data linkages to IMDB
Immediate/intermediate outcomes:
Knowledge, behaviour, integration in four areas:
Information about Canada
Official Language knowledge
Labour market
Community connections
including non-clients for comparison
Pilots, experimentation 8

10. Examples of Findings: Client Needs identified in APPR Employment
Connecting with professional networks in the Canadian workplace
A desire to find employment within field of expertise
Balance between working in a survival job vs need for language training to obtain a better employment, e.g. within field of expertise
Language
Increase demand for literacy and lower level CLBs due to increase in refugees admissions
Need to obtain CLB 4 for citizenship purposes
Health and wellness
Mental health support for children, youth and adults due to trauma
“Health and Wellness” awareness
Support Service
Need for child care to facilitate participation in services, particularly language training
Translation and interpretation (outside of IRCC services)
Transportation required to participate in IRCC and non-IRCC services 9

11. Examples of Findings: Delivery Challenges from APPR 10
Lack of support services
Increased numbers of clients, waitlists
Need for more workshops and training opportunities
Service Delivery Constraints
Canadian citizens, refugee claimants
Need for assistance with immigration paperwork
Assistance with appointments outside of the SPO
Ineligible people / services
Staff turnover
Reporting requirements (including iCARE data entry)
Lack of space, facilities, funding
Administrative issues
Local labour market conditions, employment rates impact work placements
Summer, seasonal changes (holidays, school break)
External factors
Lack of promotion, marketing and outreach
Employment as a barrier for participation
Client cancellation, turnover, attendance
Client recruitment and participation
Because of higher numbers of clients, we are trying to do more group orientations in order to reduce the waiting list for some of our programs. 10

12. Examples of Findings: Clients and Services in iCARE
In , over 410,000 clients accessed at least one settlement service in Canada, a 2.7% increase in clients served over 11

13. Ongoing Performance Measurement Intermediate Outcomes
Measurement Activities at IRCC
Ongoing Performance Measurement
Settlement Services
Outputs
Immediate Outcomes
Intermediate Outcomes
Ultimate Outcomes
Monitoring
Settlement Program Design
Terms and Conditions
Objectives
Performance Measurement Strategy
Services
Audit
Planning
Inputs and Delivery
Periodic Evaluation
Employment-related Services
(ES)
Employment-related Services
(ES)
Employment-related Services
(ES)
Employment-related Services
(ES)
Employment-related Services
(ES)
Employment-related Services
(ES)
Research 12

14. How is IRCC using this information?
Planning
Identification of newcomer needs (from Needs Assessment and Referrals Services)
Design of services e.g.; Community Connections: Volunteer Coordination and Enhancements
Analysis of service patterns (e.g.: users profile, language progression, geographic distribution)
Project and Financial Monitoring
Achievement of project service targets by organizations
Service use at various groupings (local, provincial, national)
Financial monitoring
Performance Measurement
Performance reporting to Canadians, reports to Parliament
Reporting to SPOs, posting of data (Open Data Portal)
Matching with financial data to assess use of funds
Annual performance reports on Settlement Program
Support for briefings, Cabinet documents, Standing Committees; UN, OECD input
Evaluation
Assessment of outcomes, especially for clients but also indirect services
Research
Emerging needs
Factors for social, economic and cultural integration
Client vs non-clients analysis 13

15. Data Collection Pros Cons Gaps Pros, Cons, Gaps
Wealth of robust quantitative data on program activities and outputs
Extensive client profile and settlement services data
Data shared with SPOs (monthly reports) and the public (Open Data)
Rich sources of qualitative data on project delivery
Cons
Data entry challenges by service providers (time consuming)
Potential overlap in some information collected by various tools (e.g. support services in iCARE, APPR)
Variety of information collected can be hard to analyze in a timely manner
Gaps
Indirect services (e.g. curriculum development)
Data linkages between iCARE and other datasets (e.g. IMDB)
Comparable data for ALL newcomers (client vs non-client survey) 14

16. Considerations Are we collecting the right information?
Is it collected at the right time, and at the right frequency?
How/Can we alleviate the reporting burden? By combining data? Reducing overlaps?
How do we introduce new data elements (addressing gaps) without increasing the overall burden?
How do we make reporting more meaningful?
Are we providing you information to support decision-making? Project planning? 15

17. Next Steps
Share Settlement Program Evaluation Report and recommendations
Implement Evaluation recommendations including review of reporting requirements
Review and confirm performance measurement strategy (outcomes, indicators, data sources, reporting strategy)
Establish Sector Outcomes Working Group
Implement Newcomer Outcomes Survey
Engage with sector on the common settlement vision

18. Thank you! Comments & Questions?
Anne Couillard Deputy Director, Performance Measurement Unit
Settlement and Integration Policy Branch