1. PAAB Pre-Clearance Mechanism Audit Summary

2. Audit Purpose  Assess the overall performance of the process  Evaluate the level of consistency Methodology  Audit Period: June 1 st 2013 to May 31 st 2014  Elements from Six Sigma & Standard Audit Practices o Quantitative deviations – Performance o Qualitative deviations – Consistency

3. Context Variability (Audit Period)  Submissions:6987  Manufacturers: 96  Agencies:136  Products: 600+  Therapeutic Areas:25  APS Categories:26  Reviewers:13

4. Operational Performance Turn Around  99.38 % of Response Letters are sent within 10 days  0.62% of Response Letters send after 10 days

5. Operational Performance: Revisions  36.82% of Submissions were Approved after the 1 st revision  61.22% of Submissions were Approved by the 2 nd revisions  78.16% of Submissions were Approved by the 3 rd revisions

6. Consistency How do we define “Inconsistency”?  When PAAB makes a mistake?  When PAAB is inconsistent in the pre-clearance process?  When PAAB makes a ruling that doesn’t align with previous ruling?  When PAAB makes different ruling for two current submissions?  When reviewer and the submitter disagree on an issue? Science ContentContext

7. Results There were no clear and definitive examples of inconsistencies in Reviewer based decisions.  There were differing interpretations and perspectives ⁻PAAB Code ⁻Scientific data  Examples of suspected inconsistencies are needed

8. Results Escalation Data:  Total Escalations as a percentage of the population: 0.95%  PAAB ruling upheld: 49%  PAAB/Client found a solution: 33%  PAAB ruling overturned: 13%  Client withdrew: 5%

9. Results 1.In 6 samples – differing opinion between the Client and the Reviewer  4 submissions – Client provided APS reference backup  Approved  1 submission – Client provided APS reference backup  Denied  1 submission – Client, 1 issue  Approved and 1 issue  Denied 2.14% of Samples had 2 or more Reviewers 3.14% of samples had 2 or more Client representatives 4.Reviewers missing issues during their first review:  14% of samples – Reviewer missed identifying an issue(s)  24% of samples – Change made by the Client, new reference documents or translation

10. Results 5.Factors causing delays to the processing of submissions:  4% of samples – waiting for updated or new Product Monographs or Formulary Listing Letters.  2% of samples – waiting for another APS to be approved.  The majority of initial submissions omitted the layout and/or French versions 6.22% of samples – Immediately approved 7.2% of samples – Client forgot to update the Copy Deck 8.1 sample – Client couldn’t decide how they would submit the APS (branded or unbranded)

11. Promoting Consistency What is PAAB doing to support Consistency? 1.Defined Therapeutic Teams 2.OneNote Database 3.PAAB efile System 4.Reviewer Training 5.Team Meetings

12. QUESTIONS?