Recent Trends in Fuzzy Clustering: From Data to Knowledge Shenyang, August 2009

Agenda Introduction: clustering, information granulation and paradigm shift Key challenges in clustering Fuzzy objective-based clustering Knowledge-based augmentation of fuzzy clustering Collaborative fuzzy clustering Concluding comments

Clustering Areas of research and applications: Data analysis Modeling Structure determination Google Scholar -2, 190,000 hits for “clustering” (as of August 6, 2009)

Clustering as a conceptual and algorithmic framework of information granulation Data  information granules (clusters) abstraction of data Formalism of: set theory (K-Means) fuzzy sets (FCM) rough sets shadowed sets

Main categories of clustering Graph-oriented and hierarchical (single linkage, complete linkage, average linkage..) Objective function-based clustering Diversity of formalisms and optimization tools (e.g., methods of Evolutionary Computing)

Key challenges of clustering Data-driven methods Selection of distance function (geometry of clusters) Number of clusters Quality of clustering results

The dichotomy and the shift of paradigm

Fuzzy Clustering: Fuzzy C-Means (FCM) Given data x 1, x 2, …, x N, determine its structure by forming a collection of information granules – fuzzy sets Objective function Minimize Q; structure in data (partition matrix and prototypes)

Fuzzy Clustering: Fuzzy C-Means (FCM) V i – prototypes U- partition matrix

FCM – optimization Minimize subject to (a) prototypes (b) partition matrix

Optimization - details Partition matrix – the use of Lagrange multipliers d ik = ||x k -v i || 2  –Lagrange multiplier

Optimization – partition matrix (1)

Optimization- prototypes (2) Gradient of Q with respect to v s Euclidean distance

Fuzzy C-Means (FCM): An overview

Geometry of information granules m =1.2 m =2.0m =3.5 n=1

Domain Knowledge: Category of knowledge-oriented guidance Partially labeled data: some data are provided with labels (classes) Proximity knowledge: some pairs of data are quantified in terms of their proximity (closeness) Viewpoints: some structural information is provided Context-based guidance: clustering realized in a certain context specified with regard to some attribute

Clustering with domain knowledge (Knowledge-based clustering)

Context-based clustering To align the agenda of fuzzy clustering with the principles of fuzzy modeling, the following features are considered: Active role of the designer [customization of the model] The structural backbone of the model is fully reflective of relationships between information granules in the input and output space Clustering : construct clusters in input space X Context-based Clustering : construct clusters in input space X given some context expressed in output space Y

Context-based clustering: Computing considerations computationally more efficient, well-focused, designer-guided clustering process Data structure Data structure context

Context-based clustering Context-based Clustering : construct clusters in input space X given some context expressed in output space Y Context – hint (piece of domain knowledge) provided by designer who actively impacts the development of the model

Context-based clustering: Context design Context – hint (piece of domain knowledge) provided by designer who actively impacts the development of the model. As such, context is imposed by the designer at the beginning Realization of context Designer  focus  information granule (fuzzy set) (a) Designer, and (b) clustering of scalar data in output space Context – fuzzy set (set) formed in the output space

Context-based clustering: Modeling Determine structure in input space given the output is high Determine structure in input space given the output is medium Determine structure in input space given the output is low Input space (data)

Context-based clustering: examples Find a structure of customer data [clustering] Find a structure of customer data considering customers making weekly purchases in the range [$1,000 $3,000] Find a structure of customer data considering customers making weekly purchases at the level of around $ 2,500 Find a structure of customer data considering customers making significant weekly purchases who are young no context context (compound)

Context-oriented FCM Data (x k, target k ), k=1,2,…,N Contexts: fuzzy sets W 1, W 2, …, W p w jk = W i (target k ) membership of j-th context for k-th data Context-driven partition matrix

Context-oriented FCM: Optimization flow Objective function Iterative adjustment of partition matrix and prototypes Subject to constraint U in U(W j )

Viewpoints: definition Description of entity (concept) which is deemed essential in describing phenomenon (system) and helpful in casting an overall analysis in a required setting “external”, “reinforced” clusters

Viewpoints: definition viewpoint (a,b)viewpoint (a,?)

Viewpoints: definition Description of entity (concept) which is deemed essential in describing phenomenon (system) and helpful in casting an overall analysis in a required setting “external”, “reinforced” clusters

Viewpoints: definition viewpoint (a,b)viewpoint (a,?)

Viewpoints in fuzzy clustering B- Boolean matrix characterizing structure: viewpoints prototypes (induced by data)

Viewpoints in fuzzy clustering

B- Boolean matrix characterizing structure: viewpoints prototypes (induced by data)

Viewpoints in fuzzy clustering

Labelled data and their description Characterization in terms of membership degrees: F = [f ik ] i=12,…,c, k=1,2, …., N and supervision indicator b = [b k ], k=1,2,…, N

Augmented objective function  > 0

Proximity hints Characterization in terms of proximity degrees: Prox(k, l), k, l=1,2, …., N and supervision indicator matrix B = [b kl ], k, l=1,2,…, N Prox(k,l) Prox(s,t)

Proximity measure Properties of proximity: (a)Prox(k, k) =1 (b)Prox(k,l) = Prox(l,k) Proximity induced by partition matrix U:

Augmented objective function  > 0

Two general development strategies SELECTION OF A “MEANINGFUL” SUBSET OF INFORMATION GRANULES

Two general development strategies (1) HIERARCHICAL DEVELOPMENT OF INFORMATION GRANULES (INFORMMATION GRANULES OF HIGHER TYPE) Information granules Type -1 Information granules Type -2

Two general development strategies (2) HIERARCHICAL DEVELOPMENT OF INFORMATION GRANULES AND THE USE OF VIEWPOINTS Information granules Type -1 Information granules Type -2 viewpoints

Two general development strategies (3) HIERARCHICAL DEVELOPMENT OF INFORMATION GRANULES – A MODE OF SUCCESSIVE CONSTRUCTION

Information granules and their representatives Represent v k [ii] with the use of z 1, z 2, …, z c F ii F

Representation of fuzzy sets: two performance measures Entropy measure Reconstruction criterion (error)

Expressing performance through entropy measure

Reconstruction error where Requirement of “coverage” condition

Optimization problem Form a collection of prototypes Z = {z 1, z 2, …, z c } such that entropy (or reconstruction error) is minimized while satisfying coverage criterion Min Z Q subject to Optimization of fuzzification coefficient (m) Min Z Q subject to m>1 and

Collaborative structure development (2) phenomenon, process, system… Information granules data-1data-2 data-P Information granules of higher type

Collaborative structure determination: Information granules of higher order D[1] D[2] D[P] prototypes Clustering Prototypes (higher order)

Determining correspondence between clusters (3) Clustering Prototypes (higher order) zjzj Select prototypes in D[1], D[2], …, D[p] associated with z j with the highest degree of membership

Determining correspondence between clusters (4) v i [ii] zjzj D[ii] Prototype i 0 associated with prototype z j

Family of associated prototypes Prototype i 1 in D[1] associated with prototype z j Prototype i 2 in D[2] associated with prototype z j Prototype i p in D[p] associated with prototype z j …

From numeric prototypes to granular prototypes individual coordinate of the associated prototypes: a 1 a 2 …. a p  1  2 ….  p Information granule R [0,1]

The principle of justifiable granularity: Interval representation a 1 a 2 …. a p  1  2 ….  p bd 1 0 a0a0

The principle of justifiable granularity: Interval representation a 1 a 2 …. a p  1  2 ….  p bd 1 0 a0a0

The principle of justifiable granularity: optimization criterion

Hyperbox prototypes HiHi HjHj

Interval-valued fuzzy sets and granular prototypes HiHi HjHj x

Interval-valued fuzzy sets and granular prototypes vivi x Bounds of distances determined coordinate-wise

Interval-valued fuzzy sets: membership function Upper bound Lower bound

Collaborative structure determination: Structure refinement Feedback and structure refinement

Collaborative structure determination: Structure refinement Iterate Clustering at the local level Sharing findings and clustering at the higher (global) level Assessment of quality of clusters in light of the global structure  i (U)[ii] formed at the higher level Refinement of clustering Until termination criterion satisfied

Concluding comments Paradigm shift from data-based clustering to knowledge-based clustering Accommodation of knowledge in augmented objective functions Emergence of type-2 (higher type) information granules when working with collaborative clustering