Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.

Slides:



Advertisements
Similar presentations
DECISION TREES. Decision trees  One possible representation for hypotheses.
Advertisements

Decision Tree Approach in Data Mining
With support from: NSF DUE in partnership with: George McLeod Prepared by: Geospatial Technician Education Through Virginia’s Community Colleges.
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.
Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.
Data Mining Classification: Alternative Techniques
Introduction to Boosting Slides Adapted from Che Wanxiang( 车 万翔 ) at HIT, and Robin Dhamankar of Many thanks!
Machine Learning CPSC 315 – Programming Studio Spring 2009 Project 2, Lecture 5.
Model Evaluation Metrics for Performance Evaluation
Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.
Supervised classification performance (prediction) assessment Dr. Huiru Zheng Dr. Franscisco Azuaje School of Computing and Mathematics Faculty of Engineering.
Ensemble Learning: An Introduction
Image Classification.
Three kinds of learning
1 MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING By Kaan Tariman M.S. in Computer Science CSCI 8810 Course Project.
Data Mining: A Closer Look Chapter Data Mining Strategies (p35) Moh!
Bagging LING 572 Fei Xia 1/24/06. Ensemble methods So far, we have covered several learning methods: FSA, HMM, DT, DL, TBL. Question: how to improve results?
Classification of Remotely Sensed Data General Classification Concepts Unsupervised Classifications.
Machine Learning: Ensemble Methods
For Better Accuracy Eick: Ensemble Learning
3 ème Journée Doctorale G&E, Bordeaux, Mars 2015 Wei FENG Geo-Resources and Environment Lab, Bordeaux INP (Bordeaux Institute of Technology), France Supervisor:
Ensembles of Classifiers Evgueni Smirnov
Chapter 9 Accuracy assessment in remotely sensed categorical information 遥感类别信息精度评估 Jingxiong ZHANG 张景雄 Chapter 9 Accuracy assessment in remotely sensed.
Image Classification and its Applications
Issues with Data Mining
Global land cover mapping from MODIS: algorithms and early results M.A. Friedl a,*, D.K. McIver a, J.C.F. Hodges a, X.Y. Zhang a, D. Muchoney b, A.H. Strahler.
Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.
Image Classification Digital Image Processing Techniques Image Restoration Image Enhancement Image Classification Image Classification.
Chapter 9 – Classification and Regression Trees
LOGO Ensemble Learning Lecturer: Dr. Bo Yuan
Data Mining: Classification & Predication Hosam Al-Samarraie, PhD. Centre for Instructional Technology & Multimedia Universiti Sains Malaysia.
Image Classification 영상분류
Remote Sensing Supervised Image Classification. Supervised Image Classification ► An image classification procedure that requires interaction with the.
Combining multiple learners Usman Roshan. Bagging Randomly sample training data Determine classifier C i on sampled data Goto step 1 and repeat m times.
Today Ensemble Methods. Recap of the course. Classifier Fusion
Ensembles. Ensemble Methods l Construct a set of classifiers from training data l Predict class label of previously unseen records by aggregating predictions.
CLASSIFICATION: Ensemble Methods
BAGGING ALGORITHM, ONLINE BOOSTING AND VISION Se – Hoon Park.
Lecture 3 The Digital Image – Part I - Single Channel Data 12 September
Map of the Great Divide Basin, Wyoming, created using a neural network and used to find likely fossil beds See:
Digital Image Processing
The University of Mississippi Geoinformatics Center NASA MRC RPC – 11 July 2007 Rapid Prototyping of NASA Next Generation Sensors for the SERVIR System.
Classification (slides adapted from Rob Schapire) Eran Segal Weizmann Institute.
Classification Ensemble Methods 1
Data Mining and Decision Support
Classification and Prediction: Ensemble Methods Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.
Ensemble Methods Construct a set of classifiers from the training data Predict class label of previously unseen records by aggregating predictions made.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Classification and Regression Trees
Combining multiple learners Usman Roshan. Decision tree From Alpaydin, 2010.
Dr. Chen, Data Mining  A/W & Dr. Chen, Data Mining Chapter 3 Basic Data Mining Techniques Jason C. H. Chen, Ph.D. Professor of MIS School of Business.
SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.
26. Classification Accuracy Assessment
Ensemble Classifiers.
Machine Learning: Ensemble Methods
Neural Network Architecture Session 2
Trees, bagging, boosting, and stacking
Map of the Great Divide Basin, Wyoming, created using a neural network and used to find likely fossil beds See:
University College London (UCL), UK
Data Mining Practical Machine Learning Tools and Techniques
REMOTE SENSING Multispectral Image Classification
Supervised Classification
Image Information Extraction
network of simple neuron-like computing elements
Multiple Decision Trees ISQS7342
University College London (UCL), UK
MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING
MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING
Ensemble Methods: Bagging.
Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017
Presentation transcript:

Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote Sensing of Environment

Premise of the paper: Proposes criteria for assessing algorithms for supervised land cover classification. Proposes criteria for assessing algorithms for supervised land cover classification. For land classification analysis to be operational, more automated procedures will be required. For land classification analysis to be operational, more automated procedures will be required. Land cover monitoring using remotely-sensed satellite data is becoming more common. Land cover monitoring using remotely-sensed satellite data is becoming more common. Larger volumes of data at higher quality are becoming more readily available. Larger volumes of data at higher quality are becoming more readily available. No single machine learning algorithm has been shown superior for all situations. No single machine learning algorithm has been shown superior for all situations.

Supervised classification Training stage (define useful land cover categories with spectral response patterns from training data of known cover). Training stage (define useful land cover categories with spectral response patterns from training data of known cover). Classification stage (reassign image pixels to land cover categories based on the match with defined spectral attributes). Classification stage (reassign image pixels to land cover categories based on the match with defined spectral attributes). Output stage (develop a categorized data set as maps, tables, or GIS data files). Output stage (develop a categorized data set as maps, tables, or GIS data files).

Example: Supervised classification Training dataClassified image

Objectives of this study: Compare three machine learning algorithms for supervised land cover classification Compare three machine learning algorithms for supervised land cover classification based on four criteria based on four criteria using two different data sets. using two different data sets.

Data sets 8 km AVHRR data (Advanced Very High Resolution Radiometer from NOAA) 8 km AVHRR data (Advanced Very High Resolution Radiometer from NOAA) 30 m Landsat Thematic Mapper scene (from Pucallpa, Peru area) 30 m Landsat Thematic Mapper scene (from Pucallpa, Peru area) Note: Reliable land cover classifications had been derived for both data sets based on expert knowledge (used in place of ground measurements)

1984 AVHRR data included 6 channels at 8 km resolution Two { One visible

1996 Landsat TM scene included 5 bands at 30m resolution Approximately 9000 pixels can be overlaid on the 8km AVHRR data.

8 km AVHRR data To train the classifiers To train the classifiers  Overlaid Landsat scenes on AVHRR.  Each pixel was labeled as a cover type based on interpretation of Landsat scene. To test the classification results To test the classification results  Obtained a random sample of 10,000 pixels from final classification results of a previous study (they believe their test data has a high degree of confidence).

30 m Landsat Thematic Mapper scene To train the classifiers To train the classifiers  Data were selected by sampling the results of a previous study (5958 pixels). To test the classification results To test the classification results  Date were randomly selected on an additional 12,084 pixels (although not independently derived, they were used to illustrate the evaluation criteria).

The three algorithms compared: 1. C5.0 decision tree (standard) 2. Decision tree w/ “Bagging” 3. Decision tree w/ “Boosting” Note: Bagging and boosting (2 & 3) are refinements of (1) that build multiple iterations of classifiers. They can be applied to any supervised classification algorithm.

What is a decision tree? a machine learning technique (algorithm) that analyzes data, recognizes patterns, and predicts through repeated learning instances a machine learning technique (algorithm) that analyzes data, recognizes patterns, and predicts through repeated learning instances useful when it is important for humans to understand the classification structure useful when it is important for humans to understand the classification structure successfully applied to satellite data for extraction of land cover categories successfully applied to satellite data for extraction of land cover categories

1. C5.0 decision tree predicts classes by repeatedly partitioning a data set into homogeneous subsets predicts classes by repeatedly partitioning a data set into homogeneous subsets variables are used to split subsets into further subsets variables are used to split subsets into further subsets most important component is the method used to estimate splits at each “node” of the tree most important component is the method used to estimate splits at each “node” of the tree

2. C5.0 decision tree w/“Bagging” generates a decision tree for each sample generates a decision tree for each sample a final classification result is obtained by plurality vote of the individual classifiers a final classification result is obtained by plurality vote of the individual classifiers

3. C5.0 decision tree w/“Boosting” entire training set is used to generate the decision tree with a weight is assigned to each training observation entire training set is used to generate the decision tree with a weight is assigned to each training observation subsequent decision tree iterations focus on misclassified observations subsequent decision tree iterations focus on misclassified observations a final classification result is obtained by plurality vote of the individual classifiers a final classification result is obtained by plurality vote of the individual classifiers

One of the most important criteria in selecting an appropriate algorithm: the degree of human interpretation and involvement in the classification process the degree of human interpretation and involvement in the classification process Example: supervised classification (need for time intensive collection of training data) vs. unsupervised classification (no training data). Example: supervised classification (need for time intensive collection of training data) vs. unsupervised classification (no training data).

As a result: There are always trade-offs between accuracy, computational speed, and ability to automate the classification process.

Four assessment criteria were evaluated in the study: Classification accuracy – overall, mean class, and adjusted (accounts for unequal costs of misclassification, which will vary) Classification accuracy – overall, mean class, and adjusted (accounts for unequal costs of misclassification, which will vary) Computational resources required Computational resources required Stability of the algorithms w/r/t minor variability in input data Stability of the algorithms w/r/t minor variability in input data Robustness to noise in the training data (includes random noise in input and mislabeling of cover type in training data) Robustness to noise in the training data (includes random noise in input and mislabeling of cover type in training data)

Summary: Results Accuracy is comparable between the three algorithms using two data sets. Accuracy is comparable between the three algorithms using two data sets. The Bagging and Boosting algorithms are more stable and more robust to noise in the training data. The Bagging and Boosting algorithms are more stable and more robust to noise in the training data. The Bagging algorithm is the most costly, and standard decision tree is the least costly, in terms of computational resources. The Bagging algorithm is the most costly, and standard decision tree is the least costly, in terms of computational resources.

The End thank you for listening

Accuracy Accuracy is one of the primary criteria for comparing algorithms in literature. Accuracy is one of the primary criteria for comparing algorithms in literature. Accuracy = % pixels correctly classified in the test set. Accuracy = % pixels correctly classified in the test set. In this study, all three algorithms provide fairly similar accuracies (generally within 5%). In this study, all three algorithms provide fairly similar accuracies (generally within 5%).

Computational resources Likely to be a key consideration in machine learning, where “amount of work done” is used as a measure of operations performed. Likely to be a key consideration in machine learning, where “amount of work done” is used as a measure of operations performed. Standard tree: requires less resources. Standard tree: requires less resources. Bagging: number of operations increases in proportion to number of samples used. Bagging: number of operations increases in proportion to number of samples used. Boosting: number of operations is in proportion to number of iterations used. Boosting: number of operations is in proportion to number of iterations used.

Stability of algorithm Algorithm should ideally produce stable results with minor variability in input data, otherwise, it may incorrectly indicate land cover changes when none occurred. Algorithm should ideally produce stable results with minor variability in input data, otherwise, it may incorrectly indicate land cover changes when none occurred. Variable input data can be common if training data are from same locations. Variable input data can be common if training data are from same locations. Test method: random sampling generated 10 training sets (to approximate minor variation). Test method: random sampling generated 10 training sets (to approximate minor variation). Bagging and Boosting provide more stable classification (less sensitivity to variation) than a standard decision tree. Bagging and Boosting provide more stable classification (less sensitivity to variation) than a standard decision tree.

Robustness to noise Remotely sensed data is likely to be noisy due to: signal saturation, missing scans, mislabeling, problems with sensor or viewing geometry. Remotely sensed data is likely to be noisy due to: signal saturation, missing scans, mislabeling, problems with sensor or viewing geometry. Test methods: 1) random noise in input (introduced zero values randomly to simulate missing data); 2) mislabeling of cover type in training data (assigned one class to all training pixels derived from 3 Landsat scenes). Test methods: 1) random noise in input (introduced zero values randomly to simulate missing data); 2) mislabeling of cover type in training data (assigned one class to all training pixels derived from 3 Landsat scenes). Bagging and Boosting appear substantially more robust than standard C5.0 decision tree. Bagging and Boosting appear substantially more robust than standard C5.0 decision tree.

Noise: random noise Standard C5.0 decision tree clearly has higher error rates and lower stability. Standard C5.0 decision tree clearly has higher error rates and lower stability. Bagging appears slightly more stable than boosting for the Landsat data. Bagging appears slightly more stable than boosting for the Landsat data.

Noise: mislabeling of cover type in training data Causes more problems in terms of stability for the decision tree algorithms than random noise. Causes more problems in terms of stability for the decision tree algorithms than random noise. Standard C5.0 decision tree is least stable and has the highest error of all algorithms. Standard C5.0 decision tree is least stable and has the highest error of all algorithms.

Some applications of results: These same criteria can be applied to other types of algorithms such as  Neural networks  Maximum likelihood  Unsupervised classification