1. Overview Data Mining - classification and clustering
Various distance metrics
Minkowski, Manhattan, Euclidian, Max, Canberra, Cord, and HOBbit distance
- Neighborhoods and decision boundaries
P-trees and its properties
k-nearest neighbor classification
- Closed-KNN using Max and HOBbit distance
k-clustering
- overview of existing algorithms
- our new algorithm
- computation of mean and variance from the P-trees

2. Data Mining extracting knowledge from a large amount of data
Useful Information (sometimes 1 bit: Y/N)
More data volume =
less information
Data Mining
Raw data
Information Pyramid
Functionalities: feature selection, association rule mining, classification & prediction, cluster analysis, outlier analysis

3. Classification Predicting the class of a data object
also called Supervised learning B c3 b3 a3 A c2 b2 a2 c1 b1 a1
Class
Feature3
Feature2
Feature1
Training data: Class labels are known
Classifier c b a
Sample with unknown class:
Predicted class
Of the Sample

4. Types of Classifier
Eager classifier: Builds a classifier model in advance
e.g. decision tree induction, neural network
Lazy classifier: Uses the raw training data
e.g. k-nearest neighbor

5. Clustering The process of grouping objects into classes,
with the objective: the data objects are
similar to the objects in the same cluster
dissimilar to the objects in the other clusters.
A two dimensional space showing 3 clusters
Clustering is often called unsupervised learning
or unsupervised classification
the class labels of the data objects are unknown

6. Distance Metric Measures the dissimilarity between two data points.
A metric is a fctn, d, of 2 n-dimensional points X and Y, such that     
d(X, Y) is positive definite:   if (X  Y), d(X, Y) > 0
                               if (X = Y), d(X, Y) = 0
d(X, Y) is symmetric: d(X, Y) = d(Y, X)
d(X, Y) satisfies triangle inequality: d(X, Y) + d(Y, Z)  d(X, Z)

7. Various Distance Metrics
Minkowski distance or Lp distance,
Manhattan distance,
(P = 1)
Euclidian distance,
(P = 2)
Max distance,
(P = )

8. An Example d1  d2  d For any positive integer p,Z
A two-dimensional space:
Manhattan, d1(X,Y) = XZ+ ZY = 4+3 = 7
Euclidian, d2(X,Y) = XY = 5
Max, d(X,Y) = Max(XZ, ZY) = XZ = 4
d1  d2  d
For any positive integer p,

9. Some Other Distances Canberra distance Squared cord distance
Squared cord distance
Squared chi-squared distance

10. HOBbit Similarity Higher Order Bit (HOBbit) similarity:
HOBbitS(A, B) =
A, B: two scalars (integer)
ai, bi : ith bit of A and B (left to right)
m : number of bits
Bit position:
x1: x2:
y1: y2:
HOBbitS(x1, y1) = HOBbitS(x2, y2) = 4

11. HOBbit Distance The HOBbit distance between two scalar value A and B:
dv(A, B) = m – HOBbit(A, B)
The previous example:
Bit position:
x1: x2:
y1: y2:
HOBbitS(x1, y1) = HOBbitS(x2, y2) = 4
dv(x1, y1) = 8 – 3 = dv(x2, y2) = 8 – 4 = 4
The HOBbit distance between two points X and Y:
In our example (considering 2-dimensional data):
dh(X, Y) = max (5, 4) = 5

12. HOBbit Distance Is a Metric
HOBbit distance is positive definite
     if (X = Y), = 0
     if (X  Y), > 0
HOBbit distance is symmetric
HOBbit distance holds triangle inequality

13. Neighborhood of a Point
Neighborhood of a target point, T, is a set of points, S,
such that X  S if and only if d(T, X)  r 2r T X
Manhattan Euclidian Max HOBbit
If X is a point on the boundary, d(T, X) = r

14. Decision Boundary R1 X d(A,X) A
d(B,X)
decision boundary between points A and B, is the locus of the point X satisfying d(A, X) = d(B, X) B A
Decision boundary for HOBbit Distance is perpendicular to axis that makes max distance
 > 45
Euclidian B A
Max
Manhattan
 < 45
Decision boundaries for Manhattan, Euclidean and max distance

15. Notations rc(P) : root count of P-tree P P1 & P2 : P1 AND P2
N : number of pixels
n : number of bands
m : number of bits
P1 & P2 : P1 AND P2
P1 | P2 : P1 OR P2
P´ : COMPLEMENT of P
Pi, j : basic P-tree for band i bit j.
Pi(v) : value P-tree for value v of band i.
Pi([v1, v2]) : interval P-tree for interval [v1, v2] of band i.
P0 : is pure0-tree, a P-tree having the root node which is pure0.
P1 : is pure1-tree, a P-tree having the root node which is pure1.

16. Properties of P-trees 1. a) b) 2. a) b) c) d) 3. a) b) c) d)
4. rc(P1 | P2) = 0  rc(P1) = 0 and rc(P2) = 0
v1  v2  rc{Pi (v1) & Pi(v2)} = 0
rc(P1 | P2) = rc(P1) + rc(P2) - rc(P1 & P2)
rc{Pi (v1) | Pi(v2)} = rc{Pi (v1)} + rc{Pi(v2)}, where v1  v2

17. Length of the body in bytes
P-tree Header
Header of a P-tree file to make a generalized P-tree structure
1 word
2 words
4 words
Format Code
Fan-out
# of levels
Root count
Length of the body in bytes
Body of the P-tree

18. k-Nearest Neighbor Classification
1)  Select a suitable value for k   
2) Determine a suitable distance metric
3) Find k nearest neighbors of the sample using the selected metric
4)  Find the plurality class of the nearest neighbors
by voting on the class labels of the NNs
5) Assign the plurality class to the sample to be classified.

19. Closed-KNN T is the target pixels.
With k = 3, to find the third nearest neighbor,
KNN arbitrarily select one point from the boundary line of the neighborhood
Closed-KNN includes all points on the boundary T
Closed-KNN yields higher classification accuracy than traditional KNN

20. Searching Nearest Neighbors
We begin searching by finding the exact matches.
Let the target sample, T = <v1, v2, v3, …, vn>
The initial neighborhood is the point T.
We expand the neighborhood along each dimension:
along dimension i, [vi] is expanded to the interval [vi – ai , vi+bi], for some positive integers ai and bi.
Continue expansion until there are at least k points in the neighborhood.

21. HOBbit Similarity Method for KNN
In this method, we match bits of the target to the training data
Fist we find matching in all 8 bits of each band (exact matching)
let, bi,j = jth bit of the ith band of the target pixel.
Define Pti,j = Pi,j, if bi,j = 1
= Pi,j, otherwise
And Pvi,1-j = Pti,1 & Pti,2 & Pti,3 & … & Pti,j
Pnn = Pv1,1-8 & Pv2,1-8 & Pv3,1-8 & … & Pvn,1-8
If rc(Pnn) < k,
update Pnn = Pv1,1-7 & Pv2,1-7 & Pv3,1-7 & … & Pvn,1-7

22. An Analysis of HOBbit Method
Let ith band value of the target T, vi = 105 = b
[ ] = [105, 105]
1st expansion
[ ] = [ , ] = [104, 105]
2nd expansion
[ ] = [ , ] = [104, 107]
Does not expand evenly in both side:
Target = 105 and center of [104, 111] = ( ) / 2 =
And expands by power of 2.
Computationally very cheap

23. Perfect Centering Method
Max distance metric provides better neighborhood by
- keeping the target in the center
- and expanding by 1 in both side
Initial neighborhood P-tree (exact matching):
Pnn = P1(v1) & P2(v2) & P3(v3) & … & Pn(vn)
If rc(Pnn) < k
Pnn = P1(v1-1, v1+1) & P2(v2-1, v2+1) & … & Pn(vn-1, vn+1)
Pnn = P1(v1-2, v1+2) & P2(v2-2, v2+2) & … & Pn(vn-2, vn+2)
Computationally costlier than HOBbit Similarity method
But a little better classification accuracy

24. Finding the Plurality Class
Let, Pc(i) is the value P-trees for the class i
Plurality class =

25. Performance
Experimented on two sets of Arial photographs of The Best Management Plot (BMP) of Oakes Irrigation Test Area (OITA), ND
Data contains 6 bands: Red, Green, Blue reflectance values, Soil Moisture, Nitrate, and Yield (class label).
Band values ranges from 0 to 255 (8 bits)
Considering 8 classes or levels of yield values: 0 to 7

26. Performance – Accuracy
1997 Dataset:

27. Performance - Accuracy (cont.)
1998 Dataset:

28. Performance - Time
1997 Dataset: both axis in logarithmic scale

29. Performance - Time (cont.)
1998 Dataset : both axis in logarithmic scale