Presentation is loading. Please wait.

Presentation is loading. Please wait.

Vincent Granville, Ph.D. Co-Founder, DSC

Published byFarida Sutedja Modified over 6 years ago

Similar presentations

Presentation on theme: "Vincent Granville, Ph.D. Co-Founder, DSC"— Presentation transcript:

1 Vincent Granville, Ph.D. Co-Founder, DSC

2 Feature Selection For Unsupervised Learning Abstract After reviewing popular techniques used in supervised, unsupervised and semi-supervised machine learning, we focus on feature selection methods in these different contexts, especially the metrics used to assess the value of a feature or set of features, be it binary, continuous or categorical variables. We go in deeper details and review modern feature selection techniques for unsupervised learning, typically relying on entropy-like criteria. While these criteria are usually model-dependent or scale-dependent, we introduce a new model-free, data-driven methodology in this context, with an application to an interesting number theory problem (simulated data set) in which each feature has a known theoretical entropy. We also briefly discuss high precision computing as it is relevant to this peculiar data set, as well as units of information smaller than the bit. IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

3 Content 1. Review of supervised and unsupervised learning 2. Feature selection * Supervised (goodness of fit) * Unsupervised (entropy) 3. New approach (unsupervised) IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

4 Supervised Learning Based on training sets, cross-validation, goodness-of-fit. Popular techniques: 1. Linear or logistic regression, predictive modeling 2. Neural nets, deep learning 3. Supervised classification 4. Regression and decision trees IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

5 Unsupervised Learning
No training set. Popular techniques: 1. Pattern recognition, association rules 2. Unsupervised clustering, taxonomy creation 3. Data reduction or compression 4. Graph-based methods 5. NLP, image processing 6. Semi-supervised Note: Overlap between supervised and unsupervised. Example: Neural nets (can be supervised or unsupervised.) IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

6 Star clusters classification based on age and metal content
Unsupervised Clustering: Example Star clusters classification based on age and metal content IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

7 The Concept of Feature (1/2)
Two types of variables: 1. Dependent variable (or response) – supervised learning 2. Independent variables (or predictor or feature) – usually cross-correlated Y = a1 X1 + a2 X2 + …. Here, * Y is the dependent variable, * the X’s are the features, * the a’s are the model parameters (to be estimated) IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

8 The Concept of Feature (2/2)
Variables (features) can be: * Discrete / continuous * Qualitative (gender, country, cluster label) * Mixed (unstructured, text, data) * Binary (dummy variable to represent gender) * Raw, binned, summarized, or compound variables Potential issues: * Duplicated or Missing data * Fuzzy data (when merging man-made databases, or due to typos) Example: MIT = M.I.T = Massachusetts Institute of Technology Solution: use lookup table, e.g. matching MIT with its variations IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

9 Feature Selection: Supervised Learning
Select features with best predictive power. Algorithms: * Step-wise (add one feature at a time) * Backward (start with all features, removing one at a time) * Mixed Criteria for feature selection * R-squared (or robust version of this metric) * Goodness-of-fit-metric, confusion matrix (clustering) Notes: * Unlike PCA, it leaves features unchanged (easier for interpretation) * Combinatorial problem, local optimum OK, stopping rule needed IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

10 Feature Selection: Unsupervised Learning
Select features with highest entropy. Criteria for feature selection * Shannon entropy (categorical features) * Joint entropy (measured on a set of features) * Differential entropy (for continuous features) * Akaide information (related to maximum likelihood) * Kullback-Leibler divergence (equivalent to Akaide information) Drawbacks * Model-dependent (not data-driven) * Scale-dependent IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

11 New Methodology for Unsupervised Learning (1/3)
Create an artificial response Y, as if dealing with a linear regression problem. Set regression coefficients to 1. Compute R-squared (or similar metrics) on subsets of features, to identify best subsets given a fixed number of features. Add one (or two) feature at a time. Proceed as in the supervised learning framework. Benefits * Scale-invariant * Model-free, data-driven * Simple, easy to understand * Can handle categorical features, as dummy variables More advanced version * Test various sets of regression coefficients (Monte Carlo simulations) IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

12 New Methodology for Unsupervised Learning (2/3)
Example Simulated data sets with two binary features X1 and X2, with artificial response Y = X1 + X2, and with known theoretical entropy for each feature. Features are correlated and also exhibit auto-correlation, as in real data sets. The data sets have 47 observations. Instead of entropy, we use correlation between response and a feature, to measure the amount of information attached to the feature in question. Results Computed Shannon entropy, theoretical entropy, and the correlation-based information metric introduced here, are almost equivalent: the higher the correlation, the higher the entropy. Divergence occurs only when both features almost have the same entropy, so not an issue. The best feature is the one maximizing the selection criterion. IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

13 New Methodology for Unsupervised Learning (3/3)
Details about the Example The data set in the spreadsheet consists of the first 47 digits of two numbers: * Feature #1: digits of log(3/2) in base b = * Feature #2: digits of SQRT(1/2) in base b = 2 Interesting facts * For more than 47 observations, HPC is needed due to machine precision * A digit in base 1 < b < 2 carries less than one bit of information * The theoretical entropy is proportional to the logarithm of the base b * The methodology was also successfully tested on continuous features IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

14 Resources The following web document (see URL below) contains: * References to various entropy criteria and feature selection techniques * Details about the new methodology * Access to spreadsheet with data and detailed computations * Reference to the underlying number theory context (numeration systems) * Reference to high precision computing Link: IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

15 Thank you Vincent Granville, Ph.D. Co-Founder, DSC —
DataScienceCentral.com IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

16 IBM Community Day: Data Science / July 24, 2018 / © 2018 IBM Corporation

Download ppt "Vincent Granville, Ph.D. Co-Founder, DSC"

Similar presentations

Text mining Gergely Kótyuk Laboratory of Cryptography and System Security (CrySyS) Budapest University of Technology and Economics www.crysys.hu.

Text mining Gergely Kótyuk Laboratory of Cryptography and System Security (CrySyS) Budapest University of Technology and Economics
Independent Component Analysis

Independent Component Analysis
Feature Selection as Relevant Information Encoding Naftali Tishby School of Computer Science and Engineering The Hebrew University, Jerusalem, Israel NIPS.

Feature Selection as Relevant Information Encoding Naftali Tishby School of Computer Science and Engineering The Hebrew University, Jerusalem, Israel NIPS.
Data Mining Feature Selection. Data reduction: Obtain a reduced representation of the data set that is much smaller in volume but yet produces the same.

Data Mining Feature Selection. Data reduction: Obtain a reduced representation of the data set that is much smaller in volume but yet produces the same.
Chapter 3 – Data Exploration and Dimension Reduction © Galit Shmueli and Peter Bruce 2008 Data Mining for Business Intelligence Shmueli, Patel & Bruce.

Chapter 3 – Data Exploration and Dimension Reduction © Galit Shmueli and Peter Bruce 2008 Data Mining for Business Intelligence Shmueli, Patel & Bruce.
CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.

CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.
CS 4100 Artificial Intelligence Prof. C. Hafner Class Notes March 27, 2012.

CS 4100 Artificial Intelligence Prof. C. Hafner Class Notes March 27, 2012.
Pattern Recognition and Machine Learning

Pattern Recognition and Machine Learning
Supervised Learning Recap

Supervised Learning Recap
Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.

Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.
Lecture 17: Supervised Learning Recap Machine Learning April 6, 2010.

Lecture 17: Supervised Learning Recap Machine Learning April 6, 2010.
Machine Learning CMPT 726 Simon Fraser University

Machine Learning CMPT 726 Simon Fraser University
© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.
Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)
Introduction to Data Mining Engineering Group in ACL.

Introduction to Data Mining Engineering Group in ACL.
Anomaly detection with Bayesian networks Website: John Sandiford.

Anomaly detection with Bayesian networks Website: John Sandiford.
INTRODUCTION TO MACHINE LEARNING. $1,000,000 Machine Learning  Learn models from data  Three main types of learning :  Supervised learning  Unsupervised.

INTRODUCTION TO MACHINE LEARNING. $1,000,000 Machine Learning  Learn models from data  Three main types of learning :  Supervised learning  Unsupervised.
Bayesian networks Classification, segmentation, time series prediction and more. Website: Twitter:

Bayesian networks Classification, segmentation, time series prediction and more. Website: Twitter:
Data Mining Chapter 1 Introduction -- Basic Data Mining Tasks -- Related Concepts -- Data Mining Techniques.

Data Mining Chapter 1 Introduction -- Basic Data Mining Tasks -- Related Concepts -- Data Mining Techniques.
Predictive Analytics World CONFIDENTIAL1 Predictive Keyword Scores to Optimize PPC Campaigns Vincent Granville, Ph.D. Click Forensics February 19, 2009.

Predictive Analytics World CONFIDENTIAL1 Predictive Keyword Scores to Optimize PPC Campaigns Vincent Granville, Ph.D. Click Forensics February 19, 2009.

Similar presentations

Ads by Google