Presentation is loading. Please wait.

Presentation is loading. Please wait.

SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1.

Published byRuby Mitchell Modified over 9 years ago

Similar presentations

Presentation on theme: "SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1."— Presentation transcript:

1 SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1

2 Binary Classification Suppose we have two groups for which each case is a member of one or the other, and that we know the correct classification (“truth”). We will call the two groups Disease and Healthy Suppose we have a prediction method that produces a single numerical value, and that small values of that number suggest membership in the Healthy group and large values suggest membership in the Disease group. How can we measure the success of the prediction method? First, consider the case when we have a cutoff that defines which group is predicted. February 12, 2014BST 226 Statistical Methods for Bioinformatics2

3 DiseaseHealthyTotal Predict DiseaseA (True Positive)B (False Positive)A+B Predict HealthyC (False Negative)D (True Negative)C+D TotalA+CB+DA+B+C+D February 12, 2014BST 226 Statistical Methods for Bioinformatics3 A: True Positive (TP), hit D: True negative (TN), correct rejection B: False positive (FP), false alarm, Type I error C: False negative (FN), miss, Type II error

4 DiseaseHealthyTotal Predict DiseaseA (True Positive)B (False Positive)A+B Predict HealthyC (False Negative)D (True Negative)C+D TotalA+C (Positive)B+D (Negative)A+B+C+D February 12, 2014BST 226 Statistical Methods for Bioinformatics4 Sensitivity, True Positive Rate (TPR), recall TPR = TP/P = TP/(TP+FN) = A/(A+C) Fraction of those with the Disease that are correctly predicted Specificity (SPC), True Negative Rate SPC = TN/N = TN/(TN+FP) = D/(B+D) Fraction of those Healthy who are correctly predicted Precision, Positive Predictive Value (PPV) PPV = TP/(TP+FP) = A/(A+B) Fraction of those predicted to have the Disease who do have it Negative Predictive value (NPV) NPV = TN/(TN+FN) = D/(C+D) Fraction of those predicted to be healthy who are healthy Fall-out or False Positive Rate (FPR) FPR = FP/N = FP/(FP+TN) = 1 − SPC Fraction of those healthy who are predicted to have the disease False Discovery Rate (FDR) FDR = FP/(TP+FP) = 1 − PPV Fraction of those predicted to have the disease who are healthy Accuracy (ACC) ACC = (TP+TN)/(P+N)

5 Dependence on Population February 12, 2014BST 226 Statistical Methods for Bioinformatics5 Sensitivity and Specificity depend only on the test, not on the composition of the population, other figures are dependent Sensitivity = fraction of patients with the disease who are predicted to have the disease (p = 0.98). Specificity = fraction of patients who are healthy that are classified as healthy (q = 0.99). If the population is 500 Disease and 500 healthy, then TP = 490, FN = 10, TN = 495, FP = 5 and PPV = 490/(490 + 5) = 0.9899 If the population is 100 Disease and 1000 healthy, then TP = 98, FN = 2, TN = 990, FP = 10 and PPV = 98/(98 + 10) = 0.9074 If the population is 100 Disease and 10,000 healthy, then TP = 98, FN = 2, TN = 9900, FP = 100 and PPV = 98/(98 + 100) = 0.4949

6 ROC Curve (Receiver Operating Characteristic) If we pick a cutpoint t, we can assign any case with a predicted value ≤ t to Healthy and the others to Disease. For that value of t, we can compute the number correctly assigned to Disease and the number incorrectly assigned to Disease (true positives and false positives). For t small enough, all will be assigned to Disease and for t large enough all will be assigned to Healthy. The ROC curve is a plot of true positive rate vs. false positive rate. If everyone is classified positive (t = 0), then TPR = TP/(TP+FN) = FP/(FP + 0) = 1 FPR = FP/(FP + TN) = FP/(FP + 0) = 1 If everyone is classified negative (t = 1), then TPR = TP/(TP+FN) = o/(0 + FN) = 0 FPR = FP/(FP + TN) = 0/(0 + TN) = 0 February 12, 2014BST 226 Statistical Methods for Bioinformatics6

7 February 12, 2014BST 226 Statistical Methods for Bioinformatics7 truth <- rep(0:1,each=50) pred <- c(rnorm(50,10,1),rnorm(50,12,1)) library(ROC) roc.data <- rocdemo.sca(truth,pred) plot1 <- function() { nz <- sum(truth==0) n <- length(truth) plot(density(pred[1:nz]),lwd=2,xlim=c(6,18), main="Generating an ROC Curve") lines(density(pred[(nz+1):n]),col=2,lwd=2) abline(v=10,col=4,lwd=2) abline(v=11,col=4,lwd=2) abline(v=12,col=4,lwd=2) } > plot1() > plot(roc.data) > AUC(roc.data) [1] 0.8988

8 February 12, 2014BST 226 Statistical Methods for Bioinformatics8

9 February 12, 2014BST 226 Statistical Methods for Bioinformatics9

10 February 12, 2014BST 226 Statistical Methods for Bioinformatics10 We now show the ROC curve for a rare outcome: > truth <- rep(0:1,c(990,10)) > pred <- c(rnorm(990,10,1),rnorm(10,12,1)) > plot(rocdemo.sca(truth,pred)) > AUC(rocdemo.sca(truth,pred)) [1] 0.9011111

11 February 12, 2014BST 226 Statistical Methods for Bioinformatics11 ROC Curve for Rare Outcome

12 Statistical Significance and Classification Success It is easier for a variable to be statistically significant than for the classification using that variable to be highly accurate, measured, for example, by the ROC curve. Suppose we have 100 patients, 50 in each group (say disease and control). If the groups are separated by 0.5 times the within group standard deviation, then the p-value for the test of significance will be around 0.01 but the classification will only be 60% correct. February 12, 2014BST 226 Statistical Methods for Bioinformatics12

13 February 12, 2014BST 226 Statistical Methods for Bioinformatics13

14 Statistical Significance and Classification Success If the classification is to be correct 95% of the time, then the groups need to be separated by 3.3 times the within group standard deviation, and then the p-value for the test of significance will be around essentially 0. February 12, 2014BST 226 Statistical Methods for Bioinformatics14

15 February 12, 2014BST 226 Statistical Methods for Bioinformatics15

16 February 12, 2014BST 226 Statistical Methods for Bioinformatics16 > truth <- rep(0:1,c(80,20)) > summary(glm(truth~var1,family=binomial)) Call: glm(formula = truth ~ var1, family = binomial) Deviance Residuals: Min 1Q Median 3Q Max -2.04601 -0.45586 -0.21127 -0.05413 2.11889 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -3.4727 0.6775 -5.125 2.97e-07 *** var1 1.8202 0.4038 4.508 6.55e-06 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 100.080 on 99 degrees of freedom Residual deviance: 56.222 on 98 degrees of freedom AIC: 60.222 Number of Fisher Scoring iterations: 6

17 February 12, 2014BST 226 Statistical Methods for Bioinformatics17 > pred2 <- predict(glm(truth~var1,family=binomial),type="response") > table(truth,pred2 >.5) truth FALSE TRUE 0 75 5 1 9 11 TPR = 11/20 = 0.55 SPC = TNR = 75/80 = 0.9375 PPV = 11/16 = 0.6875 NPV = 75/84 = 0.8929 FPR = 5/80 = 0.0625 FDR = 5/16 = 0.3125 > source("http://bioconductor.org/biocLite.R") > biocLite("ROC") > library(ROC) > plot(rocdemo.sca(truth,pred2)) > abline(v=0.0625,lwd=2) > abline(h=0.55,lwd=2)

18 February 12, 2014BST 226 Statistical Methods for Bioinformatics18

19 Choosing a Cutoff Suppose that missing a disease case has an implicit cost of $1000 and a false diagnosis of disease has an implicit cost of $200. Then the cost of the procedure is 1000×FN+200×FP. With a cut-off of 0.5, the estimated cost would be (1000)(9) + (200)(5) = $10,000 per 100 patients or $100 per patient. Let’s compute the cost for different cutoffs. February 12, 2014BST 226 Statistical Methods for Bioinformatics19

20 February 12, 2014BST 226 Statistical Methods for Bioinformatics20 diagcost <- function(truth,predq,costp,costn) { n <- length(predq) names(predq) <- "" cutoffs <- c(sort(predq),1) fpvec <- rep(0,n+1) fnvec <- rep(0,n+1) costvec <- rep(0,n+1) for (i in 1:(n+1)) { predb = cutoffs[i] fp <- sum(predb & !truth) fn <- sum(!predb & truth) cost <- fp*costp+fn*costn fpvec[i] <- fp fnvec[i] <- fn costvec[i] <- cost } return(data.frame(1:(n+1),cutoffs,fpvec,fnvec,costvec)) } The least cost of $4200 (vs. $10,000) is at cutoff = 0.1286845279 with 1 false negative and 16 false positives The cutoff of 0.5853639 minimizes the total errors with 2 false positives and 9 false negatives (cost $9400)

21 February 12, 2014BST 226 Statistical Methods for Bioinformatics21

22 February 12, 2014BST 226 Statistical Methods for Bioinformatics22

Download ppt "SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1."

Similar presentations

Classification.. continued. Prediction and Classification Last week we discussed the classification problem.. – Used the Naïve Bayes Method Today..we.

Classification.. continued. Prediction and Classification Last week we discussed the classification problem.. – Used the Naïve Bayes Method Today..we.
...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.

...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.
...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,
Statistical Significance and Performance Measures

Statistical Significance and Performance Measures
Learning Algorithm Evaluation

Learning Algorithm Evaluation
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.
From the homework: Distribution of DNA fragments generated by Micrococcal nuclease digestion mean(nucs) = 113.5 bp median(nucs) = 110 bp sd(nucs+ = 17.3.

From the homework: Distribution of DNA fragments generated by Micrococcal nuclease digestion mean(nucs) = bp median(nucs) = 110 bp sd(nucs+ = 17.3.
Receiver Operating Characteristic (ROC) Curves

Receiver Operating Characteristic (ROC) Curves
Lecture 16: Logistic Regression: Goodness of Fit Information Criteria ROC analysis BMTRY 701 Biostatistical Methods II.

Lecture 16: Logistic Regression: Goodness of Fit Information Criteria ROC analysis BMTRY 701 Biostatistical Methods II.
What Happens to the Performance of a Diagnostic Test when the Disease Prevalence and the Cut-Point Change? Pathological scores Healthy scores Healthy population.

What Happens to the Performance of a Diagnostic Test when the Disease Prevalence and the Cut-Point Change? Pathological scores Healthy scores Healthy population.
The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.
Logistic Regression Predicting Dichotomous Data. Predicting a Dichotomy Response variable has only two states: male/female, present/absent, yes/no, etc.

Logistic Regression Predicting Dichotomous Data. Predicting a Dichotomy Response variable has only two states: male/female, present/absent, yes/no, etc.
Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.
Model Evaluation Metrics for Performance Evaluation

Model Evaluation Metrics for Performance Evaluation
Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,

Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,
Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.

Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.
Nemours Biomedical Research Statistics April 23, 2009 Tim Bunnell, Ph.D. & Jobayer Hossain, Ph.D. Nemours Bioinformatics Core Facility.

Nemours Biomedical Research Statistics April 23, 2009 Tim Bunnell, Ph.D. & Jobayer Hossain, Ph.D. Nemours Bioinformatics Core Facility.
ROC Curve and Classification Matrix for Binary Choice Professor Thomas B. Fomby Department of Economics SMU Dallas, TX February, 2015.

ROC Curve and Classification Matrix for Binary Choice Professor Thomas B. Fomby Department of Economics SMU Dallas, TX February, 2015.
1 Logistic Regression Homework Solutions EPP 245/298 Statistical Analysis of Laboratory Data.

1 Logistic Regression Homework Solutions EPP 245/298 Statistical Analysis of Laboratory Data.
Lucila Ohno-Machado An introduction to calibration and discrimination methods HST951 Medical Decision Support Harvard Medical School Massachusetts Institute.

Lucila Ohno-Machado An introduction to calibration and discrimination methods HST951 Medical Decision Support Harvard Medical School Massachusetts Institute.

Similar presentations

Ads by Google