Knowledge Mining and Soil Mapping using Maximum Likelihood Classifier with Gaussian Mixture Models ECE539 final project Instructor: Yu Hen Hu Fall 2005 Good morning, everyone. My name is LI Xiao-Bing. I am a Ph.D. candidate from University of Science and Technology of China. Now let me show you my research activities. Jian Liu 12/13/2005

Overview This study deals with data mining from soil survey maps and soil mapping with mined soil-landscape knowledge. I’ll give the details of our proposed approaches for redundancy reduction of acoustic model, and my other research work.

Soil – landscape models Soil is a product of the interaction of surrounding environments “soil-landscape model” (Hudson, 1992) Soil can be predicated given the environments

Environmental variables Environmental factors affecting soil formation: Bedrock geology Elevation (DEM) Slope gradient 1st derivative along the steepest slope Profile curvature 2nd derivative along the steepest slope Planform curvature 2nd derivative perpendicular to contour lines

Previous Approaches & Problems Fuzzy system (Zhu 2001) Elicits knowledge from a soil scientist and represents it with arbitrary curves Assumes independence of each environmental variable ANN (Zhu 2000; Behrens 2005; Scull 2005 ) Black box knowledge representation High dimensional matrix is hard to comprehend Decision trees (Bui, 1999; Qi et.al. 2003) Knowledge extracted is crisp (typical case), no information about gradation

Proposal – Knowledge Representation GMM representation is more suitable because: Probability representation well captures the physical gradation of the phenomenon The interactions between environmental variables are taken into account by the multivariate Gaussian distribution Mixture model gives a great potential of capturing the real distribution Physically a soil type may have multiple instances.

Proposal – Maximum Likelihood Classifier P(A|Class1) = 0.8 P(A|Class2) = 0.5 A then is classified into class1 based on “Maximum likelihood” Naturally evaluates the composite effect environmental variables have on the probability of soil formation

Algorithm Training procedure: Testing procedure: Standardize feature dimensions of training set For each geology group in the training data For each soil type in the geology group Fit a GMM using EM algorithm (# of mixtures is preset, k-means is used to initialize the cluster centers) Testing procedure: Standardize feature dimensions of testing set For each sample point For each class in the corresponding geology group Calculate the corresponding likelihood based on GMM The point is classified to the class with the maximum likelihood

Case Study … Training set Testing set elevation slope gradient profile curvature planform curvature geology soil map Testing set Our objective is to reduce the model size by reducing the redundancy of acoustic model. We proposed the following approaches to reach the goal. The first one is the MCE-based dimensionality reduction to reduce the number of feature dimensions. The second one is the optimal clustering and non-uniform allocation of Gaussian kernels in feature dimension to reduce the number of feature-level parameters. The third one is the state divergence-based determination of the number of Gaussian components of each state to reduce the number of components. … elevation soil map geology

Evaluation of the GMM representation The GMM representations well capture the gradation of soil on the landscape, which complies well with expert knowledge e.g. Council at footslope e.g. Elbaville at backslope Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.

Training accuracy & testing accuracy Overall, 80% classification accuracy against testing data Increasing number of mixtures leads to higher classification accuracy at an expense of exponentially increasing storage and computational load classification accuracy (%) geology area 1 geology area 2 # of mixtures training testing 1 70.04 68.07 79.80 77.13 2 76.66 74.50 78.99 76.84 4 81.51 79.27 80.03 75.55 8 83.17 80.12 84.07 79.23 Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.

Classification Accuracy vs. # of Mixtures Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.

Mapping accuracy based on field data 64 points are correctly classified out of 83 field sample points (77%), higher than traditional manual based soil survey (usually 60%) Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework. Classification result using 8 mixtures (the dark blue areas are not mapped)

More comments Standardization of feature dimensions is very effective, -- improves mapping accuracy from 55% to 80% Preprocessing techniques such as data cleaning required by decision tree is not critical to ML because the ML classifier is not as sensitive to training errors as long as they are not of a huge amount. Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.

Conclusion GMM is suitable to represent soil-landscape knowledge ML classifier with GMMs is promising for soil knowledge mining and soil mapping Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.

Future improvement? Reduce the storage and computational load so that bigger number of mixtures can be used to improve classification accuracy Use diagonal matrix to replace full covariance matrix (after applying de-correlation to the features)? Let’s go to the first method, the MCE-based dimensionality reduction. Usually the dimensionality reduction is performed by LDA transformation which separates classes through maximizing the ratio of between-class scatter matrix and within-class scatter matrix. But is has little direct relation with the classifier’s target of minimum recognition error rate. So it results in recognition performance degradation. As MCE criterion can adjust the parameters to achieve minimum recognition error, we proposed to adjust the LDA transformation and the classification parameters simultaneously according to the MCE criterion in the DFE framework.