1. Abstract Methods Objectives Conclusions Results
Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
Abstract
Methods
Sensitivity and imprecision (S&I) are the 2 main Critical Quality Attributes (CQA) of In Vitro Diagnostic reagent, besides accuracy.
They usually require 1 to 4 weeks intensive works which limits their evaluation to important milestones.
We developed a JMP script in order to systematically assess S&I from any DOE without specific testing.
This script associate tools such as bivariate graphs and PCA (Principal Component Analysis) to make results visual and thus simplify the decision making process.
To give flexibility on the design and to speed complex calculation, the script is linked to “R”. This is transparent for the user, removing the needs for high statistical / scripting knowledge and keeping the ease of use associated with JMP which is essential to get adhesion to the process.
The overall process reduces lead time of DOE analysis from 5 to 0.5 day (by automatically managing data transformation and analysis).
It also highly improves reliability (by understanding the design space of any CQA thanks to DOE) and statistical efficiency (through more efficient estimators of imprecision /sensitivity).
This reduces failure during Verification & Validation, ultimately improving Quality and TTM (Time To Market) two majors KPI (Key Process Indicator) for development.
Develop a JMP journal that
Merge several JMP files and create new JMP files with appropriate dataset formatting.
Associate R and JMP for complex calculation.
Create graphs for easier data visualization.
Model Data to predict sensitivity and Imprecision at various levels of doses.
Objectives
Promote the use of DOE by reducing the complexity of analysis
Systematically assess S&I (sensitivity and imprecision) rather than limiting its assessment to critical milestones.
Reduce the amount of testing by assessing S&I in any experiment rather than running specific experiments.
Improve Quality by automated analysis.
Improve communication by formalizing data analysis.
Conclusions
Results
In this example we were able to assess sensitivity & imprecision on a DOE covering the main factors that are potentially impactful during reagent pack manufacturing process.
This assessment lead to a modification of the manufacturing process by
changing the pooling strategy of conjugate (collect 1:1 conjugate only, rather than usual collection of 2:1 and 1:1).
reducing the conjugate concentration from 4 ng/mL to 2 ng/mL
The JMP journal automates the process of DOE analysis for sensitivity and imprecision, as well as accuracy assessment, of any DOE.
Starting from 3 JMP files (presenting the initial dataset ) the journal
automates 23 steps
generates 16 new JMP files
perform 7 data analysis (3 DOE, 2 Bivariates graphs, 2 PCA)

2. Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
Background
Material
Reaction
The immunoassay is a competitive assay.
The analyte (molecule that we intend to measure) compete with the analog to be recognized by the Antibody.
Once the reaction is terminated the unbound material (analyte and conjugate not bound to the analog) is washed away.
A substrate is added and react with the enzyme to generate a signal.
This signal (RLU) is inversely proportional to the analyte and corresponds to the fraction of conjugate that bound to the analog which is coupled to the PMP which are retained in the instrument through magnets.
PMP (para-magnetic particules) coupled with analog
Conjugate: Antibody (Aby) coupled to an Enzyme (Enz)
Molecular weight 1:1 corresponds to 1 Aby coupled to 1 Enz
Molecular weight 2:1 corresponds to 2 Aby coupled to 1 Enz
Wash, then
add substrate
DOE inputs
Range tested
Analog concentration
PMP concentration
Conjugate Molecular weight
Conjugate concentration
Min
100
0.7
1:1 2
Max
150
1.3
2:1 6

3. Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
Initial JMP files
“RLU” dataset contains the RLU (signal) obtained on all samples (calibrators, quality controls, patient samples) for each condition.
“DOE conditions” dataset contains the description of the experimental design (inputs)
“Curvedata” dataset contains the parameters of the 4-PLC equation used to interpolate the “RLU” dataset to get the samples dose from their signal (RLU)

4. Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
CV% RLU (signal): is the variance identical for any sample type and level ? 1
Bivariate graph suggests that variance (CV% on RLU signal) is identical whatever
the sample type i.e.
Calibrators (contrived samples), ¤
Quality Controls (spiked serum samples), +
Patient samples (unaltered serum samples) x
the level of RLU signal (thus the level of dose) 1a 1b
PCA, on samples as individuals and conditions of the DOE as variables, does not identify a clear pattern suggesting that CV RLU is independent of any particular sample, especially Calibrator S0.
Note: analysis on conditions will be done through DOE analysis. 1c
Depending on the outcome of Bivariate and
PCA analysis one should consider clicking on
No groups: all samples present identical CV RLU
3 groups: the sample type lead to different CV RLU (likely due to different matrix effect leading to precipitates)
2 groups: chose which sample types are identical/different to the other ones
Consequently one should click on « Pas de groupe » button indicating that samples should be analyzed altogether to increase statistical efficiency.

5. CV% RLU (signal): is the variance identical for any condition, input ?
Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
CV% RLU (signal): is the variance identical for any condition, input ? 2 2a 3
CV% RLU (signal): pooled variance and other calculations 3a 3
Pre-selected models have been prepared covering the most usual DOE analysis i.e.
DOE on main effect
DOE on second order
Depending on the decision from step 1c, one should click on DOE 1 to DOE 3 in order to analyze the various groups previously identified.
Depending on the decision from
step 1b: is CV RLU on calibrator S0 identical or different from other calibrator levels ?
step 1c: are the various sample types leading to different CV RLU ?
step 2a: is any specific condition or input leading to different CV RLU ?
One should select how pooled variance will be calculated
For such typical experiment the confidence interval (95% 2-sided) is highly reduced
Usual CI = ( df = (4-1) = 3 )
New CI = ( df = 18*14*(2-1) = 252 )
making the conclusion much more reliable than it used to be.
For this dataset (as for most of our experiments) we conclude that variance (CV% on RLU signal) is not statistically different whatever
the inputs
the sample types
the level of RLU signal (thus the level of dose)
And then we combined all data for a more robust evaluation of pooled variance (CV RLU = 2.5%).
This makes the optimization of variance meaningful when the objective is to reduce CV% RLU from 2% to 1.5%.
DOE analysis suggests that variance (CV% on RLU signal) is not statistically different whatever the inputs i.e.
Analog concentration (on PMP),
PMP concentration,
Conjugate « Molecular weight »
Conjugate concentration

6. Prepare RLU signal for further calculation
Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France 3b
Prepare RLU signal for further calculation
From CV% RLU (step 3a), the script automatically generates signal levels (RLU) corresponding to LoB, LoD, LoQ according to CLSI guideline EP17-A2, and return extreme signal levels (S0 highest signal, S5 lowest signal) for each condition.

7. Interpolate RLU signals to get doses 4
Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
Interpolate RLU signals to get doses 4a 4 4b
RLU from steps 3b and 4a are then interpolated, thanks to 4-PLC parameters (stored in « Curvedata » dataset for each condition) in order to generate 2 new JMP files « Imprecision profile » and « DOE sensitivity »

8. Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France
Sensitivity DOE 4b
For this dataset we identified that the main drivers for sensitivity (LoB, LoD, LoQ have similar models) and imprecision are
Molecular Weight of the conjugate
Conjugate concentration
«DOE sensitivity» JMP file helps identifying conditions/inputs that are levers of sensitivity.
We usually run specific experiments to evaluate sensitivity and imprecision This takes at least 5 days (20 days for imprecision), with df = 119/269/269 for LoB/LoD/LoQ, respectively. The new process permits a rapid estimation, without any specific testing, with similar df = 252 and thus reasonable confidence interval.
In addition one gets a model indicating which factors are levers of sensitivity allowing a more thoughtful optimization.
On this project we decided to limits conjugate pooling strategy to 1:1 (MW = 1:1) and to reduce conjugate concentration.
This makes the optimization of sensitivity meaningful when the objective is to reduce, for example, LoQ from 25 to 20 pg/mL.

9. Get more with less: systematic evaluation of sensitivity and imprecision
through a JMP Script associating DOE and PCA
Emmanuel Romeu, Ounes Chadli
R&D Beckman-Coulter Diagnostic, Marseille France 4a
Imprecision profile
For this dataset the graph helps identifying that
Conjugate MW = 1:1 (blue) gives more sensitive curve shapes compare to conjugate MW 2:1 (red)
Lower conjugate concentration (circle) is beneficial compare to higher conjugate concentration (star).
« Imprecision profile » JMP file allows for an overall visualization of imprecision (either within-run or within-laboratory imprecision).
Note that usual determinations are performed at calibrator levels (vertical black lines) which do not permit an accurate estimation of imprecision at the very low end.