Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.7: Instance-Based Learning Rodney Nielsen.

Slides:

Advertisements

Similar presentations

Advertisements

Conceptual Clustering

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

Data Mining Classification: Alternative Techniques

Searching on Multi-Dimensional Data

1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.

Classification Algorithms – Continued. 2 Outline  Rules  Linear Models (Regression)  Instance-based (Nearest-neighbor)

Classification Algorithms – Continued. 2 Outline  Rules  Linear Models (Regression)  Instance-based (Nearest-neighbor)

Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)

Computational Support for RRTs David Johnson. Basic Extend.

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Clustering. 2 Outline  Introduction  K-means clustering  Hierarchical clustering: COBWEB.

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

CS Instance Based Learning1 Instance Based Learning.

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Chapter 3: Cluster Analysis  3.1 Basic Concepts of Clustering  3.2 Partitioning Methods  3.3 Hierarchical Methods The Principle Agglomerative.

DATA MINING AND MACHINE LEARNING Addison Euhus and Dan Weinberg.

Module 04: Algorithms Topic 07: Instance-Based Learning

Issues with Data Mining

DATA MINING LECTURE 10 Classification k-nearest neighbor classifier Naïve Bayes Logistic Regression Support Vector Machines.

CSC321: Neural Networks Lecture 12: Clustering Geoffrey Hinton.

Algorithms: The Basic Methods Witten – Chapter 4 Charles Tappert Professor of Computer Science School of CSIS, Pace University.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.6: Linear Models Rodney Nielsen Many of.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.3: Decision Trees Rodney Nielsen Many of.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.4: Covering Algorithms Rodney Nielsen Many.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.2 Statistical Modeling Rodney Nielsen Many.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Sections 4.1 Inferring Rudimentary Rules Rodney Nielsen.

1Ellen L. Walker Category Recognition Associating information extracted from images with categories (classes) of objects Requires prior knowledge about.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.8: Clustering Rodney Nielsen Many of these.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Classification (slides adapted from Rob Schapire) Eran Segal Weizmann Institute.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall 6.8: Clustering Rodney Nielsen Many / most of these.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 6.2: Classification Rules Rodney Nielsen Many.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall DM Finals Study Guide Rodney Nielsen.

Chapter 13 (Prototype Methods and Nearest-Neighbors )

KNN Classifier.  Handed an instance you wish to classify  Look around the nearby region to see what other classes are around  Whichever is most common—make.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.5: Mining Association Rules Rodney Nielsen.

DATA MINING LECTURE 10b Classification k-nearest neighbor classifier

Chapter 4: Algorithms CS 795. Inferring Rudimentary Rules 1R – Single rule – one level decision tree –Pick each attribute and form a single level tree.

Classification Algorithms Covering, Nearest-Neighbour.

1 Learning Bias & Clustering Louis Oliphant CS based on slides by Burr H. Settles.

Instance-Based Learning Evgueni Smirnov. Overview Instance-Based Learning Comparison of Eager and Instance-Based Learning Instance Distances for Instance-Based.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.

Data Mining CH6 Implementation: Real machine learning schemes(2) Reporter: H.C. Tsai.

Data Mining Practical Machine Learning Tools and Techniques Chapter 6.5: Instance-based Learning Rodney Nielsen Many / most of these slides were adapted.

Linear Models & Clustering Presented by Kwak, Nam-ju 1.

Data Mining Practical Machine Learning Tools and Techniques Chapter 6.3: Association Rules Rodney Nielsen Many / most of these slides were adapted from:

Rodney Nielsen Many of these slides were adapted from: I. H. Witten, E. Frank and M. A. Hall Data Science Algorithms: The Basic Methods Clustering WFH:

Data Science Practical Machine Learning Tools and Techniques 6.8: Clustering Rodney Nielsen Many / most of these slides were adapted from: I. H. Witten,

Data Mining Practical Machine Learning Tools and Techniques

k-Nearest neighbors and decision tree

Data Science Algorithms: The Basic Methods

Data Science Algorithms: The Basic Methods

Instance Based Learning

Data Science Algorithms: The Basic Methods

Data Science Algorithms: The Basic Methods

K Nearest Neighbor Classification

Data Mining Practical Machine Learning Tools and Techniques

Instance Based Learning

Classification Algorithms

CSC 558 – Data Analytics II, Prep for assignment 1 – Instance-based (lazy) machine learning January 2018.

Text Categorization Berlin Chen 2003 Reference:

Data Mining CSCI 307, Spring 2019 Lecture 21

Data Mining CSCI 307, Spring 2019 Lecture 23

Data Mining CSCI 307, Spring 2019 Lecture 11

Presentation transcript:

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.7: Instance-Based Learning Rodney Nielsen Many of these slides were adapted from: I. H. Witten, E. Frank and M. A. Hall

Rodney Nielsen, Human Intelligence & Language Technologies Lab Algorithms: The Basic Methods ● Inferring rudimentary rules ● Naïve Bayes, probabilistic model ● Constructing decision trees ● Constructing rules ● Association rule learning ● Linear models ● Instance-based learning ● Clustering

Rodney Nielsen, Human Intelligence & Language Technologies Lab Instance-Based Learning ● Distance function defines what’s learned ● Most instance-based schemes use Euclidean distance: ● x (i) and x (l) : two instances with d attributes ● Taking the square root is not required when comparing distances

Rodney Nielsen, Human Intelligence & Language Technologies Lab Normalization and Other Issues ● Different attributes are measured on different scales  need to be normalized: ● Nominal attributes: distance either 0 or 1 ● Common policy for missing values: assumed to be maximally distant (given normalized attributes)

Rodney Nielsen, Human Intelligence & Language Technologies Lab Finding Nearest Neighbors Efficiently Simplest way of finding nearest neighbor: linear scan of the data Classification takes time proportional to the product of the number of instances in training and test sets Nearest-neighbor search can be done more efficiently using appropriate data structures We will discuss two methods that represent training data in a tree structure: kD-trees and ball trees

Rodney Nielsen, Human Intelligence & Language Technologies Lab kD-tree Example

Rodney Nielsen, Human Intelligence & Language Technologies Lab Using kD-trees: Example

Rodney Nielsen, Human Intelligence & Language Technologies Lab More on kD-trees Complexity depends on depth of tree, given by logarithm of number of nodes Amount of backtracking required depends on quality of tree (“square” vs. “skinny” nodes) How to build a good tree? Need to find good split point and split direction Split direction: direction with greatest variance Split point: median value along that direction Using value closest to mean (rather than median) can be better if data is skewed Can apply this recursively

Rodney Nielsen, Human Intelligence & Language Technologies Lab kD-tree Example

Rodney Nielsen, Human Intelligence & Language Technologies Lab Using kD-trees: Example

Rodney Nielsen, Human Intelligence & Language Technologies Lab Building Trees Incrementally Big advantage of instance-based learning: classifier can be updated incrementally Just add new training instance! Can we do the same with kD-trees? Heuristic strategy: Find leaf node containing new instance Place instance into leaf if leaf is empty Otherwise, split leaf according to the longest dimension (to preserve squareness) Tree should be re-built occasionally (i.e. if depth grows to twice the optimum depth)

Rodney Nielsen, Human Intelligence & Language Technologies Lab Ball Trees Problem in kD-trees: corners Observation: no need to make sure that regions don't overlap Can use balls (hyperspheres) instead of hyperrectangles A ball tree organizes the data into a tree of d-dimensional (or k-dimensional) hyperspheres Normally allows for a better fit to the data and thus more efficient search

Rodney Nielsen, Human Intelligence & Language Technologies Lab Ball Tree Example

Rodney Nielsen, Human Intelligence & Language Technologies Lab Using Ball Trees Nearest-neighbor search is done using the same backtracking strategy as in kD-trees Ball can be ruled out from consideration if: distance from target to ball's center exceeds ball's radius plus current upper bound

Rodney Nielsen, Human Intelligence & Language Technologies Lab Building Ball Trees Ball trees are built top down (like kD-trees) Don't have to continue until leaf balls contain just two points: can enforce minimum occupancy (same in kD-trees) Basic problem: splitting a ball into two Simple (linear-time) split selection strategy: Choose point farthest from ball's center Choose second point farthest from first one Assign each point to these two points Compute cluster centers and radii based on the two subsets to get two balls

Rodney Nielsen, Human Intelligence & Language Technologies Lab Discussion of Nearest-Neighbor Learning ● Often very accurate ● Assumes all attributes are equally important ● Remedy: attribute selection or weights ● Possible remedies against noisy instances: ● Take a majority vote over the k nearest neighbors ● Removing noisy instances from dataset (difficult!) ● If n   and k/n  0, error approaches minimum ● kD-trees become inefficient when number of attributes is too large (approximately > 10) ● Ball trees (which are instances of metric trees) work well in somewhat higher-dimensional spaces