Center for Biofilm Engineering Al Parker, PhD, Biostatistician Center for Biofilm Engineering Montana State University Statistics and Biofilms June 29, 2012

Standardized Biofilm Methods Laboratory Darla Goeres Al Parker Marty Hamilton Diane Walker Lindsey Lorenz Paul Sturman Kelli Buckingham- Meyer

What is statistical thinking?  Data  Experimental Design  Uncertainty and variability assessment

What is statistical thinking?  Data (pixel intensity in an image? log(cfu) from viable plate counts?)  Experimental Design - controls - randomization - replication (How many coupons? experiments? technicians? labs?)  Uncertainty and variability assessment

Why statistical thinking?  Anticipate criticism (design method and experiments accordingly)  Provide convincing results (establish statistical properties)  Increase efficiency (conduct the least number of experiments)  Improve communication

Why statistical thinking? Standardized Methods

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

A standard laboratory method is said to be relevant to a real-world scenario if, given the same inputs, the laboratory outcome is equivalent to the real-world outcome. Relevance

Elbow Prosthesis - in vivo study

Urinary catheter in vivo study

Urinary Catheter Biofilm

CV Catheter in vivo study

Biofilm in the Catheter Tip 1,000 X magnification Sheep (control)

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

A standard laboratory method is said to be reasonable if the method can be performed inexpensively using typical microbiological techniques and equipment. Reasonableness

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Resemblance of Controls Independent runs of the method produce nearly the same control data, as indicated by a small standard deviation. Statistical tool: nested analysis of variance (ANOVA)

86 mm x 128 mm plastic plate with 96 wells Lid has 96 pegs Resemblance Example: MBEC

A100 50:NNGCSC B50 50:NNGCSC C25 50:NNGCSC D :NNGC E :NNGC F :NNGC G :NNGC H :NNGC MBEC Challenge Plate disinfectant neutralizer test control

Resemblance Example: MBEC Mean LD= 5.55 Control Data: log 10 (cfu/mm 2 ) from viable plate counts rowcfu/mm 2 log(cfu/mm 2 ) A 5.15 x B 9.01 x C 6.00 x D 3.00 x E 3.86 x F 2.14 x G 8.58 x H 4.29 x

ExpRow Control LD Mean LDSD 1A B5.95 1C5.78 1D5.48 1E5.59 1F5.33 1G4.93 1H5.63 2A B5.71 2C5.54 2D5.33 2E5.11 2F5.48 2G5.33 2H5.41 Resemblance Example: MBEC

Resemblance from experiment to experiment Mean LD = 5.48 S r = 0.26 the typical distance between a control well LD from an experiment and the true mean LD

Resemblance from experiment to experiment The variance S r 2 can be partitioned: 98% due to among experiment sources 2% due to within experiment sources

S n c m c 2 + Formula for the SE of the mean control LD, averaged over experiments S c = within-experiment variance of control LDs S E = among-experiment variance of control LDs n c = number of control replicates per experiment m = number of experiments 2 2 S m E 2 SE of mean control LD = CI for the true mean control LD = mean LD ± t m-1 x SE

8 2 Formula for the SE of the mean control LD, averaged over experiments S c = 0.02 x (0.26) 2 = S E = 0.98 x (0.26) 2 = n c = 8 m = SE of mean control LD = = % CI for the true mean control LD = 5.48 ± 12.7 x = (3.20, 7.76)

Resemblance from technician to technician Mean LD = 5.44 S r = 0.36 the typical distance between a control well LD and the true mean LD

The variance S r 2 can be partitioned: 0% due to technician sources 24% due to between experiment sources 76% due to within experiment sources Resemblance from technician to technician

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Repeatability Independent runs of the method in the same laboratory produce nearly the same outcome, as indicated by a small repeatability standard deviation. Statistical tool: nested ANOVA

Repeatability Example Data: log reduction (LR) LR = mean(control LDs) – mean(disinfected LDs)

ExpRow Control LD Mean LDSD 1A B5.95 1C5.78 1D5.48 1E5.59 1F5.33 1G4.93 1H5.63 2A B5.71 2C5.54 2D5.33 2E5.11 2F5.48 2G5.33 2H5.41 Repeatability Example: MBEC A :NNGCSC B 50 50:NNGCSC C 25 50:NNGCSC D :NNGC E :NNGC F :NNGC G :NNGC H :NNGC

Repeatability Example: MBEC Mean LR = 1.63 ExpRow Control LD Control Mean LDCol Disinfected 6.25% LD Disinfected Mean LDLR 1A B C D E F G4.93 1H5.63 2A B C D E F G5.33 2H5.41

Repeatability Example Mean LR = 1.63 S r = 0.83 the typical distance between a LR for an experiment and the true mean LR

S n c m c 2 + Formula for the SE of the mean LR, averaged over experiments S c = within-experiment variance of control LDs S d = within-experiment variance of disinfected LDs S E = among-experiment variance of LRs n c = number of control replicates per experiment n d = number of disinfected replicates per experiment m = number of experiments S n d m d 2 + S m E 2 SE of mean LR =

Formula for the SE of the mean LR, averaged over experiments S c = within-experiment variance of control LDs S d = within-experiment variance of disinfected LDs S E = among-experiment variance of LRs n c = number of control replicates per experiment n d = number of disinfected replicates per experiment m = number of experiments CI for the true mean LR = mean LR ± t m-1 x SE

Formula for the SE of the mean LR, averaged over experiments S c 2 = S d 2 = S E 2 = n c = 8, n d = 5, m = 2 SE of mean LR = = % CI for the true mean LR= 1.63 ± 12.7 x = 1.63 ± 7.46 = (0.00, 9.09)

How many coupons? experiments? n c m m n d m margin of error= t m-1 x no. control coupons (n c ): no. disinfected coupons (n d ): no. experiments (m)

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

A method should be sensitive enough that it can detect important changes in parameters of interest. Statistical tool: regression and t-tests Responsiveness

disinfectant neutralizer test control Responsiveness Example: MBEC A: High Efficacy H: Low Efficacy A100 50:NNGCSC B50 50:NNGCSC C25 50:NNGCSC D :NNGC E :NNGC F :NNGC G :NNGC H :NNGC

Responsiveness Example: MBEC This response curve indicates responsiveness to decreasing efficacy between rows C, D, E and F

Responsiveness Example: MBEC Responsiveness can be quantified with a regression line: LR = row For each step in the decrease of disinfectant efficacy, the LR decreases on average by 0.97.

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

A standard laboratory method is said to be rugged if the outcome is unaffected by slight departures from the protocol. Ruggedness

Parameters in the protocol:  Sonication Power: 130, 250, 480 watts  Sonication Duration: 25, 30, 35 minutes  Treatment Temperature: 20, 22, 24 o C  Incubation Time: 16, 17, 18 hours Ruggedness Testing of the MBEC

Ruggedness Test Design Run Incubation Time Treatment Temperature Sonication Power Sonication Duration 117 hrs22°C250 watts hrs20°C130 watts hrs24°C480 watts hrs24°C480 watts hrs24°C130 watts hrs20°C480 watts hrs20°C480 watts hrs22°C250 watts hrs20°C130 watts hrs24°C130 watts25

A100 50:NNGCSC B50 50:NNGCSC C25 50:NNGCSC D :NNGC E :NNGC F :NNGC G :NNGC H :NNGC MBEC Challenge Plate disinfectant neutralizer test control

Ruggedness Testing of the Controls

LD(controls) = (IncubationTime – 17) (SonicationDuration -30) (TreatmentTemperature – 22) (SonicationPower – 250) (BiofilmGrowth – 5.87)  All terms are small and not of practical importance inside the range of values tested  None of the model terms were statistically significant  This model allows one to quantifiably predict how deviations from the protocol affect the experimental outcome

A100 50:NNGCSC B50 50:NNGCSC C25 50:NNGCSC D :NNGC E :NNGC F :NNGC G :NNGC H :NNGC MBEC Challenge Plate disinfectant neutralizer test control

Ruggedness Testing of the LRs

LR(H) = – (IncubationTime – 17)* (SonicationDuration -30) – (TreatmentTemperature – 22) (SonicationPower – 250)  Only IncubationTime was statistically significant*  Except for IncubationTime, the terms are small and not of practical importance inside the range of values tested

Ruggedness Testing of the LRs  Only IncubationTime was statistically significant*  Except for TreatmentTemperature, the terms are small and not of practical importance inside the range of values tested LR(A) = (IncubationTime – 17)* (SonicationDuration -30) – (TreatmentTemperature – 22) (SonicationPower – 250)

Results of the ASTM ILS for the MBEC  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Collaboration

ASTM Interlaboratory Study (ILS) Process  Register test method  Conduct ruggedness testing  Minimum of 6 participating labs  Technical contact Instructions Supplies Data template  Research report  Precision & Bias statement

ASTM ILS #25570  Eight labs  Three experimental test days at each lab  Three disinfectants tested/experiment day non-chlorine oxidizer phenol quaternary ammonium compound

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Control Data

Untreated Control Variability Lab No Exp Mean LD Within plate % Among plate % Among exp day % Among lab % Repeatability SD Reproducibility SD %34%25% %27%53% %12%49% %0%83% %0%36% %7%85% %24%0% %0%49% All %11%9%76%

Attributes of a Standard Method: Seven R’s  Relevance  Reasonableness  Resemblance  Repeatability (intra-laboratory)  Responsiveness  Ruggedness  Reproducibility (inter-laboratory)

Independent runs of the method by different researchers in different laboratories produce nearly the same outcome (e.g. LR). This assessment requires a collaborative (multi- lab) study. Reproducibility

Treated Data: LR (Non-chlorine oxidizer)

Treated Data: LR (Phenol)

Treated Data: LR (Quat)

Oxidizer Results DisinfectantRowMean LR Within Among lab % Repeatability SD Reproducibility SD lab % Oxidizer A5.5075%25% B4.4196%4% C3.0392%8% D1.7285%15% E0.6050% F %66% G %0% H %0%0.5223

Phenol Results DisinfectantRowMean LR Within Among lab % Repeatability SD Reproducibility SD lab % Phenol A %0% B %0% C2.5980%20% D1.1557%43% E0.3429%71% F %48% G %44% H %0%0.3009

Quat Results DisinfectantRowMean LR Within Among lab % Repeatability SD Reproducibility SD lab % Quat A3.6436%64% B2.2635%65% C1.3446%54% D0.9527%73% E0.5826%74% F0.1850% G %22% H %0%0.3598

Repeatability at a glance …

Reproducibility at a glance …

ASTM Precision and Bias Statement Untreated Control Data Variance Assessment # of Labs # of Exps Mean LD a Sources of Variability Repeatability SD b Reproducibility SD b Within plate % Among plate % Among exp day % Amon g lab % %11%9%76% Precision and Bias 12.1 Precision: An interlaboratory study (ASTM ILS #650) of this test method was conducted at eight laboratories testing three disinfectants (non-chlorine oxidizer, phenol and quaternary ammonium compound) at 8 concentrations (depicted in Fig. 2). An ANOVA model was fit with random effects to determine the resemblance of the untreated control data and the repeatability and reproducibility of the treated data The reproducibility standard deviation was 0.67 for the mean log densities of the control biofilm bacteria for this protocol, based on averaging across eight wells on each plate. The sources of variability for the untreated control data are provided in Table 1. Table 1. Untreated control data variance assessment The repeatability (Fig. 5) and reproducibility (Fig. 6) of each disinfectant at each concentration is summarized For each of the three disinfectant types considered, the protocol was statistically significantly responsive to the increasing efficacy levels. The log reduction of the non-chlorine oxidizer increased by 0.87 for each increase in efficacy level. The log reduction of the phenol disinfectant increased by 0.87 for each increase in efficacy level. The log reduction of the quat increased by 0.5 for each increase in efficacy level Bias: Randomization is used whenever possible to reduce the potential for systematic bias.

Summary  Even though biofilms are complicated, it is feasible to develop biofilm methods that meet the “Seven R” criteria  Good experiments use control data and randomization.  Invest effort in more experiments versus more replicates (coupons or wells) within an experiment.  Assess uncertainty by SEs and CIs.  For additional statistical resources for biofilm methods, check out:

Center for Biofilm Engineering A National Science Foundation Engineerin g Research Center established in Questions?