Non-Parametric Density Functions

Slides:

Advertisements

Similar presentations

Principles of Density Estimation

Advertisements

CHAPTER 13: Alpaydin: Kernel Machines

Nonparametric Methods: Nearest Neighbors

Christoph F. Eick Questions and Topics Review Dec. 10, Compare AGNES /Hierarchical clustering with K-means; what are the main differences? 2. K-means.

Non-Parametric Density Functions Christoph F. Eick 12/24/08.

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) ETHEM ALPAYDIN © The MIT Press, 2010

Notes Sample vs distribution “m” vs “µ” and “s” vs “σ” Bias/Variance Bias: Measures how much the learnt model is wrong disregarding noise Variance: Measures.

INTRODUCTION TO MACHINE LEARNING Bayesian Estimation.

An Introduction of Support Vector Machine

INTRODUCTION TO Machine Learning 3rd Edition

K Means Clustering , Nearest Cluster and Gaussian Mixture

Lecture 3 Nonparametric density estimation and classification

1er. Escuela Red ProTIC - Tandil, de Abril, Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.

Ai in game programming it university of copenhagen Statistical Learning Methods Marco Loog.

Regression. So far, we've been looking at classification problems, in which the y values are either 0 or 1. Now we'll briefly consider the case where.

Instance Based Learning

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Pattern Recognition and Machine Learning

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

統計計算與模擬政治大學統計系余清祥 2003 年 6 月 9 日 ~ 6 月 10 日第十六週：估計密度函數

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Statistical analysis and modeling of neural data Lecture 4 Bijan Pesaran 17 Sept, 2007.

Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.

INSTANCE-BASE LEARNING

1 An Introduction to Nonparametric Regression Ning Li March 15 th, 2004 Biostatistics 277.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

CS Instance Based Learning1 Instance Based Learning.

Binary Variables (1) Coin flipping: heads=1, tails=0 Bernoulli Distribution.

Ch. Eick: Support Vector Machines: The Main Ideas Reading Material Support Vector Machines: 1.Textbook 2. First 3 columns of Smola/Schönkopf article on.

1 Lazy Learning – Nearest Neighbor Lantz Ch 3 Wk 2, Part 1.

Introduction and Motivation Approaches for DE: Known model → parametric approach: p(x;θ) (Gaussian, Laplace,…) Unknown model → nonparametric approach Assumes.

Non-Parametric Learning Prof. A.L. Yuille Stat 231. Fall Chp 4.1 – 4.3.

COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.

CHAPTER 7: Clustering Eick: K-Means and EM (modified Alpaydin transparencies and new transparencies added) Last updated: February 25, 2014.

Topic9: Density-based Clustering

Image Modeling & Segmentation Aly Farag and Asem Ali Lecture #2.

MACHINE LEARNING 8. Clustering. Motivation Based on E ALPAYDIN 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2  Classification problem:

Lecture3 – Overview of Supervised Learning Rice ELEC 697 Farinaz Koushanfar Fall 2006.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Radial Basis Function ANN, an alternative to back propagation, uses clustering of examples in the training set.

Nonparametric Density Estimation Riu Baring CIS 8526 Machine Learning Temple University Fall 2007 Christopher M. Bishop, Pattern Recognition and Machine.

Machine Learning ICS 178 Instructor: Max Welling Supervised Learning.

1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.

Cost Modeling of Spatial Query Operators Using Nonparametric Regression Songtao Jiang Department of Computer Science University of Vermont October 10,

Eick: kNN kNN: A Non-parametric Classification and Prediction Technique Goals of this set of transparencies: 1.Introduce kNN---a popular non-parameric.

Debrup Chakraborty Non Parametric Methods Pattern Recognition and Machine Learning.

CHAPTER 8: Nonparametric Methods Alpaydin transparencies significantly modified, extended and changed by Ch. Eick Last updated: March 4, 2011.

Density Estimation in R Ha Le and Nikolaos Sarafianos COSC 7362 – Advanced Machine Learning Professor: Dr. Christoph F. Eick 1.

Lecture 7 Spring 2010 Dr. Jianjun Hu CSCE883 Machine Learning.

Spectral Methods for Dimensionality

Probability Distributions

More on Clustering in COSC 4335

Lecture #26 Thursday, November 17, 2016 Textbook: 14.1 and 14.3

Instance Based Learning

INTRODUCTION TO Machine Learning 3rd Edition

Ch8: Nonparametric Methods

Machine Learning Basics

Nonparametric Density Estimation

Roberto Battiti, Mauro Brunato

K Nearest Neighbor Classification

Statistical Learning Dong Liu Dept. EEIS, USTC.

INTRODUCTION TO Machine Learning

LECTURE 16: NONPARAMETRIC TECHNIQUES

Parametric Methods Berlin Chen, 2005 References:

Nonparametric density estimation and classification

政治大學統計系余清祥 2004年5月26日~ 6月7日第十六、十七週：估計密度函數

SVMs for Document Ranking

Presentation transcript:

Non-Parametric Density Functions Christoph F. Eick 8/21/2015

Non-Parametric Density Estimation Goal is to obtain a density function: http://en.wikipedia.org/wiki/Probability_density_function Parametric (single global model), semiparametric (small number of local models) Nonparametric: similar inputs have similar outputs Keep the training data;“let the data speak for itself” Given x, find a small number of closest training instances and interpolate from these Aka lazy/memory-based/case-based/instance-based learning

Influence Functions The influence function of a data object is a function fBy : Fd → R0+ which is defined in terms of a basic influence function fB Examples of basic influence functions are:

Example: Kernel Density Estimation D={x1,x2,x3,x4} fDGaussian(x)= influence(x1) + influence(x2) + influence(x3) influence(x4)= 0.04+0.06+0.08+0.6=0.78 x1 x3 0.04 0.08 y x2 x4 0.06 x 0.6 Remark: the density value of y would be larger than the one for x

Notations O={o1,…,on} is a dataset whose density has to be measured r is the dimensionality of O n is the number of objects in O d: FrX Fr→ R0+ is a distance function for the objects in O; d is assumed to be Euclidian distance unless specified otherwise dk(x) is the distance of x its k-nearest neighbor in O

Gaussian Kernel Density Functions Density functions fB: defined as the sum of the influence functions of all data points. Given N data objects, O={o1,…, on} the density function is defined as—normalized version: Non-Normalized Version: Example: Gaussian Kernel Non-parametric Density Function f Remark:  is called kernel width; called h in our textbook!

Density Functions for different values of h/ Remark:

kNN-based Density Functions Example: kNN density function with variable kernel width—width is proportional to dk(x) in point x: the distance of the kth nearest neighbor in O to x; is usually difficult to normalize due to the variable kernel width; therefore, the integral over the density function does not add up to 1 and usually is . Parameter selection: Instead of the width , k has to be selected! or Alpaydin mentions the following variation: