Performance measures Morten Nielsen, CBS, Department of Systems Biology, DTU.

Slides:



Advertisements
Similar presentations
©2011 Elsevier, Inc. Molecular Tools and Infectious Disease Epidemiology Betsy Foxman Chapter 8 Determining the Reliability and Validity and Interpretation.
Advertisements

CENTER FOR BIOLOGICAL SEQUENCE ANALYSISTECHNICAL UNIVERSITY OF DENMARK DTU Project in Immunological Bioinformatics Morten Nielsen, CBS, BioCentrum, DTU.
Performance measures Morten Nielsen, CBS, Department of Systems Biology, DTU.
Kin 304 Regression Linear Regression Least Sum of Squares
CENTER FOR BIOLOGICAL SEQUENCE ANALYSISTECHNICAL UNIVERSITY OF DENMARK DTU T cell Epitope predictions using bioinformatics (Neural Networks and hidden.
Regression, Correlation. Research Theoretical empirical Usually combination of the two.
Sensitivity, Specificity and ROC Curve Analysis.
Artificial Neural Networks 2 Morten Nielsen BioSys, DTU.
Artificial Neural Networks 2 Morten Nielsen Depertment of Systems Biology, DTU.
Regression and Correlation
CS 8751 ML & KDDEvaluating Hypotheses1 Sample error, true error Confidence intervals for observed hypothesis error Estimators Binomial distribution, Normal.
CAP and ROC curves.
Performance measures Morten Nielsen, CBS, BioCentrum, DTU.
Inferring Gene Regulatory Networks from Asynchronous Microarray Data David Oviatt, Dr. Mark Clement, Dr. Quinn Snell, Kenneth Sundberg Department of Computer.
Lecture 4: Correlation and Regression Laura McAvinue School of Psychology Trinity College Dublin.
Accuracy of Prediction How accurate are predictions based on a correlation?
Quantitative Business Analysis for Decision Making Simple Linear Regression.
ROC & AUC, LIFT ד"ר אבי רוזנפלד.
Determine whether each curve below is the graph of a function of x. Select all answers that are graphs of functions of x:
ROC Curve and Classification Matrix for Binary Choice Professor Thomas B. Fomby Department of Economics SMU Dallas, TX February, 2015.
Correlation Coefficients Pearson’s Product Moment Correlation Coefficient  interval or ratio data only What about ordinal data?
Lecture 5 Correlation and Regression
Chapter 4 Two-Variables Analysis 09/19-20/2013. Outline  Issue: How to identify the linear relationship between two variables?  Relationship: Scatter.
Evaluation – next steps
Chapter 14 – Correlation and Simple Regression Math 22 Introductory Statistics.
Irkutsk State Medical University Department of Faculty Therapy Correlations Khamaeva A. A. Irkutsk, 2009.
1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.
Selection of T Cell Epitopes Using an Integrative Approach Mette Voldby Larsen cand. scient. in Biology PhD in Immunological Bioinformatics.
Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =
MEASURES of CORRELATION. CORRELATION basically the test of measurement. Means that two variables tend to vary together The presence of one indicates the.
Biostatistics Lecture 17 6/15 & 6/16/2015. Chapter 17 – Correlation & Regression Correlation (Pearson’s correlation coefficient) Linear Regression Multiple.
Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.
Slide 8- 1 Copyright © 2010 Pearson Education, Inc. Active Learning Lecture Slides For use with Classroom Response Systems Business Statistics First Edition.
Regression Regression relationship = trend + scatter
4.4 Factoring Quadratic Expressions P Factoring : Writing an expression as a product of its factors. Greatest common factor (GCF): Common factor.
Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU IIB-INTECH, UNSAM, Argentina.
Chapter 11 Correlation and Simple Linear Regression Statistics for Business (Econ) 1.
Factoring General Trinomials Factoring Trinomials Factors of 9 are: REVIEW: 1, 93, 3.
Inferential Statistics. The Logic of Inferential Statistics Makes inferences about a population from a sample Makes inferences about a population from.
Chapter 4 Summary Scatter diagrams of data pairs (x, y) are useful in helping us determine visually if there is any relation between x and y values and,
5.4 Line of Best Fit Given the following scatter plots, draw in your line of best fit and classify the type of relationship: Strong Positive Linear Strong.
STA302: Regression Analysis. Statistics Objective: To draw reasonable conclusions from noisy numerical data Entry point: Study relationships between variables.
Pearson Correlation Coefficient 77B Recommender Systems.
Pattern Classification an Diagnostic Decision Yongnan Ji.
Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.
Linear Prediction Correlation can be used to make predictions – Values on X can be used to predict values on Y – Stronger relationships between X and Y.
1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.
Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.
V. Rouillard  Introduction to measurement and statistical analysis CURVE FITTING In graphical form, drawing a line (curve) of best fit through.
Simple Linear Regression The Coefficients of Correlation and Determination Two Quantitative Variables x variable – independent variable or explanatory.
Testing Predictive Performance of Ecological Niche Models A. Townsend Peterson, STOLEN FROM Richard Pearson.
STA302: Regression Analysis. Statistics Objective: To draw reasonable conclusions from noisy numerical data Entry point: Study relationships between variables.
ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.
A statistical model for predicting risk of re-imprisonment
Data Analytics CMIS Short Course part II Day 1 Part 4: ROC Curves Sam Buttrey December 2015.
Performance Indices for Binary Classification 張智星 (Roger Jang) 多媒體資訊檢索實驗室 台灣大學 資訊工程系.
Regression Line  R 2 represents the fraction of variation in the data (regression line)
Model Evaluation Saed Sayad
Regression and Correlation of Data Correlation: Correlation is a measure of the association between random variables, say X and Y. No assumption that one.
©2004 by Pearson Education. ©2004 by Pearson Education.
Performance Measures II
©2004 by Pearson Education. ©2004 by Pearson Education.
The receiver operating characteristic (ROC) curve
Image Classification via Attribute Detection
The Least-Squares Line Introduction
ROC Curves and Operating Points
Ch 4.1 & 4.2 Two dimensions concept
Evaluation of assay performance with positive and negative binary reference sets Evaluation of assay performance with positive and negative binary reference.
Evaluating Classifiers
RANDOM NUMBERS SET # 1:
Presentation transcript:

Performance measures Morten Nielsen, CBS, Department of Systems Biology, DTU

Performance measures Meas Pred

Pearsons’ correlation coefficient

Mean Squared Error

Quantitative data Accuracy of prediction method Sort

Matthews correlation - Threshold of 0.5 False negative True positive True negative False positive

Matthews correlation - Threshold of

Performance measure – Roc curve False negative True positive True negative False positive

Performance measure – Roc curve False negative True positive True negative False positive

ThresholdTPFNTP/(TP+FN)FPTNFP/(FP+TN) > > > > > AUC = 0.5 AUC = 1.0 ROC curves

AUC (area under the ROC curve)

Summary MCC and PCC Random = 0.0 Perfect = 1.0 (or -1) ROC (AUC) Random = 0.5 Perfect = 1 (or 0)