1. Ontario Colleges of Applied Arts and Technology
Key Performance Indicators (KPI)
April 28, 2017

2. “A performance indicator is a policy relevant statistic, number or qualitative description that provides an indication that the university, some aspect of it, or the university system is performing as it should.”
Association of Universities and Colleges of Canada

3. Provincial Impetus for Action
The Provincial Auditor's Report in 1996 recommended that the ministry improve the accountability of colleges through performance measurement. The report recommended the ministry:
set measurable goals for the colleges;
develop way to track and report performance in achieving goals;
report publicly on performance and establish funding policies that contribute to achievement of stated goals.
Independent of the ministry, and at the same time, ACAATO (now Colleges Ontario) was exploring performance measurement for colleges.

4. Setting the Stage
January 1996, discussion paper “Accountability in Learning Centred Environment” released by ACAATO.
November 1996, ministry - ACAATO working group established to advance work of discussion paper.
February 1997, the Deputy Minister, in an address to ACAATO, invited the college system to work with the ministry to implement performance indicators.
April 1997, ACAATO discussion paper “An Accountability Framework for Ontario’s Community Colleges” was released to colleges for comment.
During the summer of 1997, colleges and ministry began working on this initiative ultimately identifying five Key Performance Indicators.
December 1997, three college-ministry working groups were established to define the Key Performance Indicators and develop an implementation plan.
December 1997, the government approved performance-based funding and the collection and reporting of performance measures.

5. Goals and Objectives
The ministry established two goals for the college system:
Excellence: to offer high quality programs that meet the needs of students and employers.
Accountability: to demonstrate excellence through the achievement of specific outcomes.
Five performance objectives identified by ministry state the outcomes colleges are expected to accomplish to achieve the goal of excellence:
Student Success (Retention)
Student Satisfaction
Graduate Satisfaction
Graduate Employment
Employer Satisfaction
These objectives were consistent with the objectives the colleges identified in the ACAATO report.

6. Terms of Reference & Guiding Principles
The three working groups established in December 1997 were asked to:
Recommend definitions for the five indicators;
Design an instrument or procedure to collect indicator data;
Propose an implementation plan to collect, tabulate and report; and
Recommend a method to measure performance.
The following principles guided the three working groups:
Data should be relevant to ministry’s goals and objectives for postsecondary education and the colleges’ Accountability Framework objectives;
Data collection and tabulation should be fair, consistent, objective and auditable;
Build on colleges’ expertise, existing material and consider experiences of other jurisdictions; and
Build on ministry’s procedures and activities related to performance indicators.

7. Working Groups
The working groups were assigned to report on one of the following:
Post College Outcomes and Graduate Satisfaction
Employer Satisfaction
Student Satisfaction and Student Retention
Membership included broad representation from among colleges (faculty, administration, career placement officers, etc), college student governments, college alumni association, ACAATO and ministry.
Each working group was chaired by a college staff member.
Majority of members in each working group were from colleges.
Each working group included a wide spectrum of expertise and responsibilities at colleges, ACAATO or ministry.

8. Defining Appropriate Data to Capture
The three working groups were asked to:
Recommend a precise definition for the five KPI
Identify and explicitly define the appropriate data element(s) used to calculate the five KPI
The working groups agreed that:
Focus should be on outcomes
The working groups believed that:
The data gathered should be of value to colleges in identifying strengths, or areas of concern, to develop strategies for improvement; and therefore strongly supported a common survey framework that contained more data elements than required for the five KPI
Colleges also be given an opportunity to ask five college specific questions which would not be reported to the ministry

9. Developing a Process to Manage Data Collection
Everyone involved agreed there was a need for:
Accuracy and validity of data.
Auditable data and process.
Level playing field: all colleges treated equally.
Recommendations:
Impartial third party collection and tabulation of data.
Graduate outcomes and Employer Satisfaction to be done by a telephone survey to maximize response rate
Student Satisfaction to be paper based
Census approach rather than a sample
For the graduate outcomes and employer satisfaction surveys, service provider required by contract to have at least a 70 % completion rate
Survey periods of fixed duration
Process to be piloted at a minimum of four colleges with different demographics, including one French-language college
Compliance with the Freedom of Information and Protection of Privacy Act

10. Assessing and Incorporating Data Integrity Rules
In developing process to manage data collection and tabulation, strong emphasis was placed on data integrity rules.
Graduate Outcomes Surveys
Data sent by colleges to service provider are audited by public accountant, confirming that the graduate:
Completed all requirements to graduate
Graduated from an eligible program
Graduated in an eligible semester
Interviewer verifies that person surveyed is graduate
Process audited by public accountant to ensure stated internal control objectives are achieved, including
Validity of completed surveys
Accuracy of information recorded by interviewer
Accuracy in calculating survey results
Objectivity and non-bias of survey results
Graduate outcomes survey KPI results are audited

11. Data Integrity Rules, con’d.
Student Satisfaction Survey
Standard administrative protocol used to conduct survey to ensure consistent collection of data
Local agreement between college and student government(s) formalizes the internal survey administration
Survey Administrators confirm that the survey process was conducted according to the instructions
Service Provider
Verifies integrity of incoming packages containing surveys
Checks 600 randomly selected surveys for accuracy
Verifies that separately submitted student profile data is within range of survey results
Service provider’s process audited by public accountant to ensure stated internal control objectives are achieved
Student Satisfaction KPI result is audited
Graduation Rate
Graduates to have been reported in final semester in the audited enrolment report

12. Consensus Reached: the Report
In a short time frame, each of the three working groups was able to reach consensus on most issues.
The three working groups prepared a single report Implementation of Key Performance Indicators for Ontario’s Colleges of Applied Arts and Technology.
Report approved by ACAATO’s General Assembly, April 17, 1998.
Report submitted to Ministry, April 28, 1998.
Report’s recommendations approved by Minister, summer 1998.
Minister announced the introduction of Key Performance Indicators, August 28, 1998.
Data collection began in Fall 1998.

13. Ontario Colleges’ KPI
Graduate Employment*: per cent of graduates who sought employment after graduation, found work.
Graduate Satisfaction*: per cent of graduates who were satisfied that their college education was useful in achieving their goals after graduation.
Employer Satisfaction*: per cent of employers satisfied with the college preparation of graduates
Student Satisfaction: per cent of students satisfied with the quality of their learning experience, support services and educational resources.
Graduation Rate: per cent of students who graduate within approximately 200% of the normal duration of their program of instruction, except for applied degrees where a seven-year time frame applies.
* Tied to funding

14. Collecting the KPI
Graduate Satisfaction and Graduate Employment KPIs: Source - Graduate Outcomes Survey
Survey is conducted and results tabulated by independent service provider Forum Research Inc.
Survey is conducted six months after graduation.
3 survey periods per year, each about 8 weeks in duration.
Employer Satisfaction KPI: Source - Employer Survey
Survey is conducted after completion of Graduate Outcomes survey.
3 survey periods per year, each about 4 weeks in duration.
Student Satisfaction KPI: Source - Student Satisfaction and Engagement Survey
Survey is in class and conducted by colleges.
Distribution and collection of questionnaires, and tabulation of responses is done by an independent service provider CCI Research Inc.
Formal survey in February, 2 weeks in duration. Advance surveys in June and November
Graduation Rate KPI: Source - college student records
Calculated by colleges using a system wide prescribed methodology.

15. The Survey Instruments
Graduate 6 months
Telephone based (with web-based option for hard-to-find graduates).
On average takes about 15 minutes to complete.
Provides more than 125 fields of data.
Employer 6months
Telephone based.
On average takes about 10 minutes to complete.
Provides more than 50 fields of data.
Student Satisfaction and Engagement
Paper based.
On average takes about 30 minutes to complete.
Provides more than 70 fields of data.
Graduate 2 years
On average takes about 14 minutes to complete.
Provides more than 100 fields of data

16. The Survey Instruments – New in 2014-15
Graduate Outcomes and Satisfaction 2 Years (3 Year Pilot)
A new follow-on annual telephone-based survey conducted 2 years after graduation.
A test of the pilot included Winter 2012 graduates from six volunteer colleges occurred in May-June The objective of the test was to assess question wording and survey flow.
A pilot with all 24 colleges began in September 2014 and surveyed graduates through end of July 2015.
The pilot is continuing and is currently surveying graduates.
In designing the years, it was deemed important to align the questions to those asked at 6 month.
Data from the two surveys, at 6 month and at 2 years, are linked at the individual graduate record.

17. Breadth of Data Collection, 2016-17
Graduate Outcomes
More than 41,700 graduate interviews were completed out of about 99,400 graduates.
42% completion rate.
About 11.9% of graduates refused to complete the survey.
About 18.9% of contact leads were found to be “dead”.
About 27.1% of graduates were not reached.
Graduates’ consent rate to the Employer Satisfaction Survey was about 19.8%.
Employer Satisfaction
About 3,700 employer interviews were completed out of about 6,800 employers.
56.1% completion rate.
About 4% of employers refused to complete the survey.
About 24% of contact leads were found to be “dead”.
About 15% of employers were not reached.
Student Satisfaction
About 148,480 surveys were processed of which more than 131,700 were eligible for inclusion in KPI.
60% of students represented in survey.

18. Public Reporting of KPI
College level KPI data are announced annually to the public and posted on the Colleges Ontario (CO) website
and subsequently on the ministry’s website
Once data becomes available, colleges are required to post KPI graduation rates, KPI employment rates and Ontario Student Assistance Program default rates by program on their websites.
The ministry’s Employment Profile (EP) website is refreshed annually. The EP includes program-level data from the graduate outcomes survey, student satisfaction and engagement survey and graduation rate dataset. The website helps inform decision- making by prospective college students, parents and guidance counsellors. The website also includes helpful career-planning links such as Ontario Job Futures, Ontario Prospects and Ontario Skills passport, and a link to the Ontario Colleges Application Service (at the program level).

19. Employment Profile (EP) – Flagship Publication
Empowers students and families to make informed decisions about postsecondary career choices. Fulfills the recommendation made in the Provincial Auditor's Report to report publicly on college performance.

20. Performance Funding
Performance funding was introduced in to support the ministry’s vision of high quality education, with an increased focus on accountability.
Allocations are dependent on the results of the institution’s performance funding indicators, weighted by funded enrolment.
Institutions are eligible for performance funding if their results are no more than 10% below the system average.
Since , with rare exceptions, every college has annually received an allocation for each of the three funded performance indicators.
Colleges have received $16.4M in performance funding annually since
Performance funding KPIs:
Employment Rate at Six Months
Graduate Satisfaction Rate
Employer Satisfaction Rate

21. KPI Administration: Steering Committees
The KPI project is overseen two joint ministry and college committees:
KPI Policy Committee chaired by the Director of the Postsecondary Accountability Branch; and
KPI Operations Subcommittee chaired by the Manager of the Colleges Unit
Committees provides advice and makes recommendations to the ministry on the administration of the KPI initiative.
Committees includes:
Student representatives (1)
Colleges Ontario nominated college representatives (10)
Ministry staff (3)
Non voting - Colleges Ontario (1) and OCAS (1)
Committees meets 4 to 5 times a year, in person or by teleconference.
Day-to-day administration is carried out by Colleges Unit staff.
Each college has a local coordinator(s) of the project.

22. KPI Administration: Operating Procedures
Annually, the ministry issues operating procedures for the administration of KPI
The relevant documents are:
Student Satisfaction and Engagement
Graduate and Employer Surveys
Graduation Rate
OSAP Program Indicators

23. Commitment to Continuous Improvement
College KPI staff, service providers and ministry are committed to continuous improvement.
Student groups, colleges, service providers and ministry staff provide Steering Committee with input.
As a result, the KPI process, now in its 19th survey year, continues to evolve. For example:
Since 2005 depersonalized KPI data have been made available to institutional researchers through a data warehouse.
A summer Student Satisfaction Survey window was introduced in 2007.
Additional questions were added to the graduate outcomes survey in 2007 focusing on further education outcomes.
Student Satisfaction Survey expanded to include “engagement” questions in 2009
Co-op Diploma Apprenticeship (CODA) programs became subject to KPI in 2012
In-class apprentices have an opportunity to participate in KPI since 2012
Braille and AODA on-line student survey introduced in 2012
Graduate Outcomes 2 years added in 2014
Service providers have introduced numerous enhancements.
A ministry-hosted workshop is held annually to examine issues, exchange best practices, etc.

24. Summary
Since its implementation, performance measurement and performance-driven funding of colleges has provided:
college students with an opportunity to influence their learning environment by way of regular feedback;
college administration and faculty with a vehicle to judge the quality of their programs, facilities, resources and make necessary improvements to better their performance; and
employers with systematic input regarding current and future workplace needs.
As a result of more qualitative analysis about colleges and their programs (the results are released into the public domain),
prospective students are better able to make informed choices as to where they might wish to study;
the Ministry of Advanced Education and Skills Development is better informed to make policy decisions; and
the taxpayers are better informed about the performance of the college sector and the investment they have made in postsecondary education.

25. Appendices Appendix A – KPI in Postsecondary Education
Appendix B – KPI Results

26. Appendix A - KPI in Postsecondary Education
Colleges*
Universities
PCCs**
Graduation Rate
Graduation Rate
Graduation Rate
Employment Rate
Employment Rate
Employment Rate
OSAP Default Rate***
OSAP Default Rate***
OSAP Default Rate***
Employer Satisfaction Rate
Employer Satisfaction Rate
Graduate Satisfaction Rate
Graduate Satisfaction Rate
Student Satisfaction Rate
Employment Rate in the Field of Study
Indicators in grey comprise the three OSAP Performance Indicators that all OSAP-eligible PSE institutions are required to publish on their institutional website at the program level.
* Includes Other Public Institutions
** Includes Other Private Institutions
***OSAP Default Rate is not technically considered a KPI.

27. Appendix B- KPI Results
College Key Performance Indicators
Graduation Year*
Year of data collection (1)
Graduate Employment (at six months)
Graduate Satisfaction
Employer Satisfaction
Student Satisfaction (2)
Graduation Rate
87.7
79.7
92.0
75.1
57.4
88.0
80.5
92.7
76.3
58.5
89.3
81.6
92.1
77.8
60.1
90.1
82.0
92.6
77.4
63.3
90.5
82.8
93.1
77.9
64.9
88.9
82.7
93.3
78.4
64.6
84.8
79.8
93.0
65.0
83.0
79.1
93.2
76.1
64.2
78.9
92.8
76.8
83.6
80.0
93.4
77.1
83.4
80.1
92.2
65.5
84.0
88.1
76.2
65.7
80.3
91.4
66.7
78.8
91.2
76.5
66.6
* Graduation Year is not relevant to Student Satisfaction, which is a survey taken of students at all levels in-year.
1. Year relates to survey cycle, for example survey cycle data released in April 2015.
2. As a result of a survey review, questions on the Student Satisfaction and Engagement Survey were revised in Comparison to previous years' results should be made with caution.