1. FDA and Pharmaceutical Manufacturing Research Projects Jeffrey T. Macher Jackson A. Nickerson Co-Principal Investigators

2. Presentation Overview  Executive summary  Project goals  Data collection and synthesis  Analysis methodology  Findings  Development opportunities and constraints

3. Executive Summary  We develop statistical models that predict the:  Probability of a facility being chosen for inspection.  Effect of investigator training, experience, and individual effects on the probability of investigational outcomes.  Characteristics and identities of facilities that correlate with the probability of non-compliance.  We present initial results for each of these analyses.  We identify additional opportunities and next steps to create value along with some constraints.

4. FDA Research Project History  Research project idea emerged in Fall, 2001.  Approached FDA in late Spring, 2002.  Formalized relationship with FDA in Fall, 2003.  Began receiving data September, 2004.

5. FDA Research Project Goals  Risk-based assessment of FDA cGMP outcomes.  Identify underlying ability of investigators and their training.  Identify underlying compliance of each facility.  Identify attributes (currently recorded by the FDA) that impact inspection outcomes.  Transfer “learning” to FDA.

6. Progress to Date  Just as new drugs go through  Discovery  Development and  Commercialization….  Our model and this presentation concludes the discovery phase of our project.  Please think of our model as a “platform” that can be developed to assess a variety of compliance issues.

7. FDA Project Approach  Compile and link FDA databases.  Estimate the likelihood of various outcomes:  NAI, VAI, OAI; Warning Letters; Field Alerts; Product Recalls.  based on…  compound/product, facility, firm, FDA district, investigator and training derived factors.  in order to …  evaluate the allocation of investigational resources.  inform effectiveness of investigator training and management.

8. FDA Databases  DQRS (Field alerts)  EES  FACTS (Inspections) – CDER only  Product Listing  Product Recalls  Product Shortages  Facility Registration (DRLS)  ORA Training database  Warning letter database

9. Data Preparation  Started with FACTS (1990-2003).  Manufacturing facilities only.  Assembled investigator training database:  Identified corporate ownership by plant by year and firms operating at a specific facility each year.  Constructed facility-year data  Added observations for years NOT inspected.  Corrected FEI/CFN mismatches.  Constructed numerous other variables.

10. Some basic “facts” about the FDA data  Years covered: FY 1990-2003  Total number of facilities inspected: 3753  Total number of “Pac codes”: 38,341  Total number of “Inspections”: 14,162  Total number of investigators: 783

11. 0 500 1,000 198919901991199219931994199519961997199819992000200120022003 Number of FDA Facility Visits per Year: 1990-2003

14. Empirical Methodology  Inspection  Probability of choosing a facility to inspect.  Detection  Probability of a non-compliance inspection outcome.  Noncompliance  Probability of noncompliance, inspection, and detection.  Detection control estimation.

15. Inspection  Groups of variables:  Technology variables RxPrompt ReleaseExt or Delayed Rel Gel CapSoft Gel CapOintment LiquidPowderGas ParenteralLg. Vol. Parent.Aerosol BulkSterileSuppositories  Industry variables Vitamins (IC 54) Necessities (IC 55) Antibiotics (IC 56) Biologics (IC 57)  Inspection decision variables Ln(Days between inspections) Surveillance = reason for inspection (0 = Compliance) Last inspection outcome (1 = OAI, 0 = NAI, VAI)  Years 1992-2003 (binary variables for each year)

16. Inspection: Explained Variance  R 2 Cumulative R 2 Technology variables 12% 12% Industry variables 9 21 Inspection Decision variables 20 51 Year dummy variables ~0 51 Omitted categories: Human Drugs (IC 60-66), select technologies, Year dummies 1990-91. Foreign inspection included in analysis but uniquely identifies many inspections and is dropped from the analysis.  Probit analysis of decision to inspect.

17. Rx0.13** Promp Rel.-0.19** Ext/del Rel.-0.19** Gel Cap-0.25** Soft Gel Cap-0.36** Ointment-0.32** Liquid-0.30** Powder-0.37** Technology Variables: Change in Probability of Inspection Gas-0.68** Parenteral-0.32** Lg Vol Parent.-0.08+ Aerosol-0.26** Bulk-0.37** Sterile-0.07** Suppositories-0.23** ** 99% confidence interval * 95% confidence interval +90% confidence interval Omitted categories: Not Classified, Bacterial antigens, Bacterial vaccines, Modified bacterial vaccines, Blood serum, Immune serum.

18. Industry and Inspection Variables: Change in Probability of Inspection Antibiotics (IC 56)0.19** Vitamins (IC 54)0.11** Necessities (IC 55)-0.06** Biologics (IC 57)-0.07** Industry VariablesInspection Variables Ln(Days btwn Insp)-0.28** Surveillance-0.84** Last outcome0.13** Omitted category: Human drugs ** 99% confidence interval * 95% confidence interval +90% confidence interval

19. Days Between Inspections Probability of Inspection Years Since Last Inspection

20. Detection  Groups of variables  Technology  Industry  Training Total training days prior to inspection (other than 5 main drug courses) Drug course 1: Basic drug school Drug course 2: Advanced drug school Drug course 3: Pre-approval inspections Drug course 4: Active Pharmaceutical Ingrediant Mfg. Drug course 5: Industrial sterilization  Investigator Experience Number of inspections in the prior 12 months Number of inspections in the prior 12-24 months  ORA District Office  Investigator Classification A consolidation of position classifications

21. Detection: Explained Variance  R 2 Cumulative R 2 Technology variables 0.9 % 0.9 % Industry variables 0.3 1.2 Training and Experience vars. 0.3 1.5 Office and Position variables 1.4 2.9 Investigator effect 4.2 7.1  Probit analysis of decision to inspect.

22. Training and Experience Variables: Change in Probability of Detection 1 1 Without investigator fixed effects. Total training days prior to inspection (less 1-5)-2.2E-03 Drug course 1: Basic drug school0.07* Drug course 2: Advanced drug school-0.05 Drug course 3: Pre-approval inspections-0.23** Drug course 4: Activ. Ingred. Mfg.-0.15* Drug course 5: Industrial sterilization0.08* No. of inspections in the prior 12 months4.8E-03+ No. of inspections in the prior 12-24 months-1.4E-03

23. ORA Office and Classification Variables: Change in Probability of Detection 2 ORA LOS0.07+ ORA KAN-0.06+ ORA NYK-0.07* ORA SJN-0.09** ORA SRL-0.10* ORA ATL-0.10** ORA DAL-0.10** ORA SAN-0.11** ORA DET-0.13** ORA NWE-0.15** All other ORA off. insignificant. Compliance0.04 Microbiologist-0.02 Investigator-0.04 Chemist-0.05 Eng/Sci-0.07 Dist/Reg. Admin.-0.10+ FDA Bureau-0.15* Technician-0.18 ORA Office VariablesPosition Variables 2 With investigator fixed effects.

24. 425 Investigators

25. Non-compliance  Detection Control Estimation  Relatively new procedure used in academic literature.  Used for assessing tax evasion, EPA compliance, and other applications.  FDA application more complicated than other applications.  Assume three actors:  Facility decides level of compliance.  Inspection decision-maker chooses when to inspect.  Investigator chooses detection or not.  Estimate all three processes simultaneously.

26. Non-compliance model  Assume inspection decisions are non-random.  Assumption is different from other applications.  Construct a likelihood function that models the probabilities of:  a plant being selected for inspection and  the outcome of the inspection.

27. Constructing a Likelihood Function L 1i = 1 L 1i = 0 L 2i = 1 L 3i = 1 L 2i = 0 The likelihood that facility i is non- compliant The likelihood that facility i is compliant The likelihood that facility i is inspected The likelihood that facility i is not inspected The likelihood that facility i is found non-compliant The likelihood that facility i is found compliant L 3i = 0

28. Likelihood Function  Three probabilities are combined to form the function:  Probability that a non-compliant facility is inspected and detected: L 1i =1, L 2i =1, L 3i =1  Probability of inspecting and not detecting noncompliance: probability that the facility is compliant: L 1i =0, L 2i =1 probability that noncompliance goes undetected: L 1i =1, L 2i =1, L 3i =0  Probability that a facility is not inspected in a given year: L 2i =0

29. “Simple” Likelihood Function LL = log { F(x 1i  1 ) G(x 2i  2 ) H(x 3i  3 ) } + log { G(x 2i  2 ) [ F(-x 1i  1 ) + F(x 1i  1 ) H(-x 3i  3 ) ] } + log { G(-x 2i  2 ) } Where A = facilities inspected and found noncompliant B = facilities inspected and found compliant C = facilities not chosen for inspection

30. Estimating the Likelihood Function  Select covariates associated with non-compliance, selection, and detection.  Non-compliance: facility-related characteristics.  Selection: factors currently used in selecting facilities.  Detection: investigator-related factors.  Use a maximum likelihood estimation to find coefficient estimates that maximize the function.  Initialize parameter estimates with results from inspection and detection analyses.

31. Change in Probability of Non-compliance Rx-0.10*-0.09-0.05-0.04 Prompt rel.0.070.08-0.13 Ext/Del rel.0.17+0.21+0.130.14 Gel cap0.20+0.19+0.050.06 Soft gel cap-7.E-050.02-0.04 Ointment0.110.08-0.18-0.15 Liquid0.21*0.22+-0.04-0.03 Powder4.E-03-0.01-0.26-0.22 Gas-0.240.150.410.36 Parenteral0.14 -0.04-0.01 Lg. vol Parent.-0.24+-0.25-0.26-0.27 Aerosol0.08 0.11-0.07 Bulk-0.18**-0.15+-0.24+-0.27+ Sterile0.09 0.030.01 Suppositories0.12 -0.26-0.27 Number of obs. 81570553712245617499

32. Vitamins0.070.17 Necessary0.130.12 Antibiotics0.23**0.22* Biologics-0.050.06 No. Thera. Classes/Plant2.E-03-3.E-03 No. Products/Plant-2.E-03-1.E-03 No. Dose forms/Plant-4.E-03-0.01 No. D.F. Routes/Plant-3.E-040.00 No. Sponsor Appl./Plant0.02* ** Ownership change (t=0)0.16 Ownership change (t=1)-0.13 Ownership change (t=2)-0.09 Ownership change (t=3)0.34+ Firms per plant-0.07 InspectionTechnologyYes Plant SelectNoYes No DetectionTrainingYes No. of obs81570553712245617499

33. “Facility-fixed” Effects  Construct binary variables for the facilities with the Greatest number of inspections.  Re-estimate non-compliance model using binary variables for these 50 facilities.  Identify those facility more or less likely than average to be non-compliant.

34. Predicted Level of Facility Non-compliance For 50 Most Inspected Facilities 28 41 45 25 35 4 33 38 13 42 14 43 30 37 10 39 50 49 46 22 17 31 12 40 1 34 26 47 8 36 21 32 18 23 2 44 3 5 19 16 9 29 20 15 7 27 Statistically more noncompliant than the mean facility. Statistically not different from the mean facility. Statistically more compliant than the mean facility.

35. Immediate Implications  Inspection and Non-compliance  New suggestions for inspection choices. Use non-compliance analysis to assess risk of any given facility, firm, or technology. –Increase focus on particular facilities and attributes. –Ownership changes.  Mixed strategy inspection plan.  Detection  Use detection analysis to assess quality of investigators and their training.  Focus investigator activities to build and maintain short-run experience.

36. Broader Implications  Our statistical methods provide a test-bed for asking and answering management and oversight questions.  Further development is needed.  DCE has potentially broad applicability to CDER and other centers at the FDA including CBER, food, etc..  What facilities are most at risk of non-compliance? Base-line non-compliance Technology Ownership changes, etc.  What manufacturers are more/less prone to non-compliance.  DCE has implications for the type, format, and processing of data to be collected and analyzed.

37. Development Opportunities  Additional variables can and are being constructed to examine additional issues.  Recall, shortages, supplement filings.  More fine-grain information on technology, manufacturing knowledge, organizational capabilities.  Evaluate manufacturer data collected in our study.  More heavily weight more recent investigations.  Expand to full set of investigators and facilities (requires additional computational resources).  Evaluate endogeneity concerns.

38. Development Constraints  Software/computer limitation.  Data preparation/man-power.  Funding resources are nearly exhausted.  Teaching.

39. Current Plan  Document current progress in a white paper.  Further develop data in hand (EES, Shortages, etc.).  We received cooperation from the gold sheets.  Work with you to develop plan for transferring results to FDA.  Look for additional funding sources.