1. 2007 Annual Conference Measuring Regulatory Excellence: Key Performance Indicators Bruce G. Matthews, P.Eng. Professional Engineers Ontario

2. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Overview Project Definition and Scope What is a Key Performance Indicator? Project Process Challenges List of Potential KPIs Survey re: Relevance and Importance Recommended KPIs Aftermath

3. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Project Definition and Scope Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator” Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.”

4. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Project Definition and Scope con’t Literature review re: best practices Review of other regulators’ performance measurement practices Survey of stakeholders re: perceptions and expectations Assess availability of data for ongoing tracking / reporting of KPIs

5. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation What is a Key Performance Indicator? A metric used to reflect the critical success factors of an organization Can be financial or non-financial Nature of KPIs will differ depending upon the nature of an organization A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe)

6. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Effective KPIs… Should help define: –The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments? –The How – Are we doing things right? Do we have the right inputs in terms of resources and processes?

7. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Project Process Literature Review Environmental Scan - Review performance measurement practices of other professional regulators Identify potential KPIs Survey stakeholders re: relevance and importance of potential KPIs Review pros and cons of potential KPIs

8. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Literature Review re: Best Practices Performance measurement best practices –Balanced scorecard –UK National Consumer Council –“The Regulatory Craft” –“Red Tape to Smart Tape” International Conference (2002)

9. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Balanced Scorecard Developed by Harvard professors Kaplan and Norton Four categories –Customers –Internal Business Processes –Financial –Organizational Capabilities Weakness re: definition of “customer”

10. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation UK National Consumer Council Three steps to credible regulation –Objectives –Content –Structure and Governance www.ncc.org.uk

11. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation “The Regulatory Craft” Book by Malcolm K. Sparrow (Brookings Institution Press, 2000) Controlling risks, solving problems and managing compliance Matrix of “Regulatory Competencies” and “Business Results” Acknowledges “internal” versus “external” measures

12. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Red Tape to Smart Tape Conference Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes www.smarttape.ca

13. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Environmental Scan Other engineering regulators in Canada Other professional regulatory bodies in Ontario and across Canada Most report key transactional statistics (e.g., number of licence applications) Little evidence of outcome measurement

14. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Challenges What does it mean to be an effective regulator? Inability to benchmark due to lack of reporting by other “comparable regulatory bodies” Can’t use measures typical for professional associations KPIs must be evaluated in context

15. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation List of Potential KPIs Competence programs Intrusions on regulatory mandate Complaints about licence holders or unlicensed individuals Licensing turnaround times Compliance action statistics Council evaluation

16. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation List of Potential KPIs con’t Fairness & dignity survey Complaints about PEO Employee engagement Percentage of licence holders with no reported violations Public confidence survey score Ratio of administrative overhead to program delivery cost

17. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Survey re: Relevance and Importance PEO Council, Senior Management and selected external stakeholders Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator

18. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Survey re: Relevance and Importance Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period Survey results were assessed in view of the best practices review –Need balance of internal and external measures) –Need to ensure key regulatory competencies are represented

19. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Recommended KPIs Competence programs Complaints about licence holders or unlicensed individuals Compliance action statistics Employee engagement Public confidence survey score

20. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Competence Programs Pros –Competent members reflect a well regulated profession –Competency is fundamental to professional regulation and easily understood by the public Cons –PEO currently has no competence programs in place and none planned in the near term

21. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Complaints about Licence Holders etc. Pros –Easily measured (transaction statistics already reported) Cons –Frivolous or vexatious complaints could skew the data –Does complaint volume really speak to regulatory effectiveness?

22. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Compliance Action Statistics Pros –Clear statement of enforcement activity –Some statistics already reported Cons –Need to define what qualifies as a “compliance action” –Of greater interest to licence holders than to the public

23. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Employee Engagement Pros –Regular survey of employees already in place –Survey is widely recognized and used Cons –Is PEO willing to take action in deficient areas? –Are the survey areas relevant to a professional regulatory organization?

24. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Public Confidence Survey Score Pros –Solid external measure of how PEO is perceived –Survey would be conducted by an independent research firm Cons –Frequency of measurement –Who is “the public”?

25. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Implementation Issues Currently no competence programs in place at PEO Measurement data exists for only three of the recommended KPIs Public Confidence Survey still in development Issues of KPI reporting (timing, frequency, format) still to be determined

26. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Aftermath Project report received and considered by PEO Council in November 2006 Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators” Concern that recommended KPIs would drive policy and program development Concern about lack of competency programs

27. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Aftermath con’t PEO Council struck a task force to further examine the question of KPIs for PEO Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs Status as of September 2007: awaiting new project charter / terms of reference

28. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Speaker Contact Information Bruce G. Matthews, P.Eng. Manager, Complaints & Discipline Professional Engineers Ontario 25 Sheppard Avenue West, Suite 1000 Toronto, Ontario M2N 6S9 T: 416-840-1076 F: 416-224-9974 E: bmatthews@peo.on.ca

29. 2007 Annual Conference Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice Jan Robinson Registrar & CEO College of Physiotherapists of Ontario September 8, 2007

30. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Interesting times accountability transparency movement of internationally educated professionals public perspective

31. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Politics in Ontario

32. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Ontario Regulators for Access (ORA) Info Sharing Best practices Collaboration

33. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation “measures” ➪ government driven

34. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation report card to progress report

35. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Key Factors Included # of internationally trained applicants (annual comparisons) top five source countries types of license granted project initiatives Focus on “10”

36. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Thomson review

37. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Key Recommendations fair registration practice code information for applicants criteria for decisions reasonable costs

38. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation alternative evidence for unattainable documents training third party review self audits with reporting appeal procedures

39. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 2002 – ISO 17024 General requirements for bodies operating certification schemes for persons

40. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation ANSI SCC

41. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 2006 Fair Access to Regulated Professions Act

42. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Key features ➪ access centre ➪ Fairness Commissioner ➪ audit

43. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Consider self audit tool gap analysis demonstrate commitment continuous improvement achieve best practice

44. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Focus on entry to practice

45. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Conversion to standards 38 – registration 10 - organization

46. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Examples The College has policies and procedures related to confidentiality and privacy The College has clear processes for reviewing, evaluating and making changes to registration requirements

47. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Examples The College has uniform competence requirements The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant.

48. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Format standard definition evidence analysis recommendation ISO reference

49. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Key Findings identified no major gaps or concerns links between College and third party hard to evaluate Identified new areas for policy work activity

50. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation identified areas where evidence weak tedious how often necessary?

51. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Key Outcomes confidence in fairness and transparency of process evidence of performance all included in one place celebration of excellence

52. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Next steps develop work plan to implement recommendations share with government and other regulators

53. Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Speaker Contact Information Jan Robinson Registrar & CEO College of Physiotherapists of Ontario e-mail : jrobinson @collegept.org www.collegept.org