So Far……  Clustering basics, necessity for clustering, Usage in various fields : engineering and industrial fields  Properties : hierarchical, flat, iterative, hard, soft, disjunctive  Types: Supervised and unsupervised  K-means algorithm – for unsupervised clustering  Vector quantization – for supervised clustering  CONTINUE with fuzzy and neural net algorithms – supervised clustering

Topics for today  Basics of fuzzy systems, and fields of application  Residual analysis methods  Fuzzy C-means algorithm – Matlab illustration  Artificial neural network – basics  Structure and working of artificial neuron  Hyper plane analysis of the output

FUZZY LOGIC SYSTEMS IN SUPERVISED CLUSTERING

Fuzzy Systems  Last decade increase in Fuzzy system implementation  More popular in the field of control systems and pattern recognition  Consumer products (washing machine,camcorders, palm pilot…) and industrial systems (to provide decision support and expert system with powerful reasoning capabilities bound by a minimum of rules)  Classical set (non-fuzzy) – either belongs to or does not belong to the set (crisp membership)  Fuzzy set – allows degree of membership for each element to range over the unit interval [0, 1]

 Fuzzy membership represent similarities of objects to imprecisely defined properties whereas, probabilities convey information about relative frequencies  Probability : some kind of likelihood or degree of certainty or if it the outcome of clearly defined but randomly occurring events  Major feature of fuzzy : expresses the amount of ambiguity in human thinking  When to use fuzzy logic ?  Continuous phenomenon, not easily breakable into discrete segments  Cannot model a process

Residual Analysis  Fuzzy reasoning :IF-THEN reasoning based on the sign of the residual Ex: IF residual-1 is positive and residual-2 is negative THEN fault1 is Present IF residual-1 is zero and residual-2 is zero THEN system is fault free :  Fuzzy Clustering : each data point belongs to all classes with a certain degree of membership. The degree is dependant upon the distance to all cluster centers. For fault diagnosis, each class could correspond to a particular fault

(1) Fuzzy centers computed by minimizing the following partition formula: Fuzzy C-means algorithm subject to C - Number of clusters N - Number of data points - fuzzy membership of the k-th point to the i-th cluster - Euclidean distance between the data point and the cluster center fuzzy weighting factor which defines the degree of fuzziness of the results (normally chosen m =2, to get analytical solution) -

(2)The cluster centers v, (centroids or prototypes) are defined as the fuzzy weighted center of gravity of the data, (3)The minimization of the partition functional (1) will give the following expression for the membership (4) Euclidean distance defined as :

Two steps of Fuzzy Clustering  Off-line phase: Learning phase, determines cluster centers of the classes (this is done by iteratively calculating membership function). A learning data set is necessary, which must contain residuals for all known faults.  On-line phase: Calculates the membership degree of the current residuals to each of the known classes.

1. Choose the number of classes C, ; Chose m=2, Initialize ( start with some arbitrary values for cluster centers and corresponding partition matrix values ) 2. Calculate the cluster centers using Eq. in (2) 3. Calculate new partition matrix using Eq. in (3) 4. Compare and. If the variation of the membership degree, calculated with an appropriate norm, is smaller than a given threshold, stop the algorithm, otherwise go back to step 2. Off-line phase Cluster centers determination

Matlab – fuzzy logic tool box – illustration of fuzzy C-means algorithm

 For the incoming data, calculate the degree of membership to all the centers using the following formula On-line phase - fuzzy membership of the k-th point to the i-th cluster - Euclidean distance between the data point and the cluster center

Artificial Neural Net systems

 Based on low level microscopic biological models  Originated from modeling of human brain and evolution  Collective behavior of NN, like a human brain, demonstrates the ability to learn, recall and generalize from training patterns of data  Consists of large number of highly interconnected processing elements (nodes)  Application areas: speech recognition, speech to text conversion, image processing, investing, trading……

 Model specified by three basic elements: Models of the processing element Models of interconnections and structures (network topology) Learning rules (the way information is stored in the network)  Each node collects values from all its input connections, performs a predefined mathematical operation and produces a single output  Net input to the node = integration function (typically dot product) combining all inputs  Each input connection is weighed. These weights (can be positive or negative values) are determined during learning process. Because of this adjustments, NN is able to learn.  Activation output of node = activation function ( net input ), usually a non-linear function

Ex: Activation function Unipolar ( output takes the value 0 or 1 ) Output y = 0, if sum < threshold (b) = 1,if sum > threshold (b) sum is dot product of input vector x= ( ) and weight vector w = ( )

Some activation functions (unipolar and bipolar)

 Input vector x = ( )  Linearly combined with weights  Then s is activated by a threshold function T(-) to produce the output y = T(s) = 1 when s > 0, else y = T(s) = -1.  Then all the input vectors x such that forms a Hyperplane H in the input vector space. H partitions the feature vector space into right and left half spaces, H+ (when sum is > b) and H- (when sum < b) The Perceptron as Hyperplane separator, where b is the threshold

Ex : consider a single perceptron with two inputs Let w1 = 2 andw2 = -1, b=0, then 2x1 - x2 = 0 determines H the points (0,0) and (1,2) belong to H The feature vector x = (x1,x2) = (2,3) is summed into S = 2(2) - 1(3) = 1 > 0, so that the activated output is y = T(1) = 1 (corresponds to H+ in the plane, i.e right half) (x1,x2) = (0,2) activates the output y = T(2(0) - 1(2)) = T(-1) = -1, which indicates that (0,2) is in the left halfspace H-. The figure shows these points.

Mapping in Hyperplane. (example of linear mapping between input and output

Non-linear mapping between input and output Example : XOR logic function or 2- bit parity problem. N = 2 inputs, M = 1 output, and Q = 4 sample vector (input/output) pairs for training, and K= 2 clusters (even and odd).

Hyperplane diagram for 2-bit parity problem

XOR function implementation in three layered network Take: result is two parallel hyperplanes that yield three convex regions. The hyperplanes are determined by

The threshold at the first neuron in the hidden layer yields The threshold at the second hidden neuron yields

hyperplanes yield three convex regions. The four sets of above inputs yield the three unique vectors Corresponding to three regions in the hyperplane

Hyperplanes showing three regions for 2-bit parity problem Regions 1 and 3make up the odd parity (Class 2),while Region 3 is even parity (Class 1).

For the second layer (output layer), the equations are as follows: Choosing Threshold = 1/2