1. Institute for Advanced Studies in Basic Sciences – Zanjan Mahdi Vasighi Supervised Kohonen Artificial Neural Networks

2. Data Analysis: Linear transformations and data reduction techniques like Principal Component Analysis (PCA) Advantage: 1. Projection from high dim. onto a low dim. coordinate system. 2. Does not require a high level of modelling expertise. Disadvantages: 1. Assuming the topology of the input data can be reduced in a linear fashion. 2. Outliers disturb the quality of the projection. 3. Visualization power deteriorates considerably if the number of relevant dimensions in the multivariate space remains high after a PCA analysis. Introduction

3. An alternative is nonlinear mapping: This explicitly aims to map objects in a low- dimensional space (usually two-dimensional) in such a way that the distances between objects are preserved in the mapping. The Kohonen maps (self organizing maps) incorporates in an unsupervised way the topology present in the data. The unsupervised problem means that one deals with a set of experimental data which have no specific associated answers (or supplemental information) attached.

4. A supervised modelling technique is able to capture the relationship between the input data (measurements, observations) and output data (properties of interest).  MLR, PLS, for regression problems  LDA for classification problems These modelling techniques fail if:  Nonlinearity or topologically incoherent relationship and Considerable number of outliers. ANNs and SVMs can tackle such nonlinear relationships in a convincing way. However, the visualization and interpretation of these models is severely hard due to the fact that they are more or less ‘black-box’ techniques. X (input) Y (output)

5. Kohonen Artificial Neural Networks The Kohonen network is probably the closest of all artificial neural networks architectures and learning schemes to the biological neuron network As a rule, the Kohonen type of net is based on a single layer of neurons arranged in a two- dimensional plane having a well defined topology A defined topology means that each neuron has a defined number of neurons as nearest neighbors, second-nearest neighbor, etc.

6. W Input vector Output Weight vector Similarity is the basis of selection of the winner neuron. In other words, there is a competition between neurons for winning. (competitive learning)

7. The Kohonen learning concept tries to map the input so that similar signals excite neurons that are very close together.

8. Top Map After the training process accomplished, the complete set of the training vectors is once more run through the KANN. In this last run the labeling of the neurons excited by the input vector is made into the table called top map. e XSXS d c b a Trained KANN e db c a Top Map

9. Weight Map The number of weights in each neuron is equal to the dimension m of the input vector. Hence, in each level of weight only data of one specific variable are handled. Trained KANN 00000 10000 11000 43110 56211 13012 32213 21123 12101 32112 X S Input Vector LLLL LLL H HHHH HHH Top Map

10. Applications of Kohonen ANNs Sampling from large amount of data Disease Diagnosis QSAR & QSPR Analysis of genomics data Making model using corrupt object (missing values) Typical similarity criteria Specific similarity criteria Number of missing values

11. Counter Propagation network (CPN) CPANN has the same structure as Kohonen network with an additional output layer with same layout as input layer.

12. Output layer Input Kohonen layer Input Target Input Similarity map Winner Based on the location of the winning unit in the input map (i.e., the unit which is most similar or closest to the presented object X), the input map and the output map are updated simultaneously at the same spatial locations. If the CPN network is trained, it can be used for prediction. Simply, an unknown input object is presented to the network. The position of the winning unit in the input map then is used to look-up the predicted value of the corresponding unit in the output map.

13. Applications of CP-ANN Building predictive calibration and classification models Spectrum Class Membership Molecular Descriptor Medicinal Activity Process Descriptor Process Condition Reaction Descriptor Reaction Mechanism

14. Detection of faulty condition in a process

15. CPN was not able to model the inverse relationship between the output and the input.  The output properties are not involved in the formation of the directing Kohonen input map. Hence, the CPN model cannot be considered as being a true supervised method.  One can easily look inside the driving input layer and relationship with output. in a CPN network the flow of information is directed from the input layer units towards the output layer. For this reason, we prefer to denote the CPN as being a pseudo- supervised strategy.

16. Supervised Kohonen network (SKN) In a SKN network, the input layer and the output layer are ‘glued’ together, thereby forming a combined input-output layer. Because in a SKN information present in the objects X and Y is used explicitly during the update of the units in the map, the topological formation of the concatenated map is driven by X and Y in a truly supervised way.

17. After training, the input and output maps are decoupled. Then, for a new input object its class membership is estimated according to the procedure outlined for the CPN network. The variables of the objects X and Y in the training set must be scaled properly, but it is not trivial how to deal with the relative weight of the number of variables in X and the number of variables in Y during the training of a SKN network.  User must determine beforehand the proper balance between the influence of the input and output objects: in general, correct scaling of the input and output variables is of utmost importance.

18. Output layer Input Kohonen layer Input Target Output Similarity map Input Similarity map Fused Similarity map Winner By using a ‘fused’ similarity measure based on a weighted combination of the similarities between an object X and all units in the input layer, and the similarities between the corresponding target and the units in the output layer, the common winning unit for both maps is determined. α(t) decreases linearly in epochs Chemo. Int. lab. 83 (2006) 99–113 XY-Fused networks

19. Simulated data sets The first data set, referred to as Odd–Even, contains 400 rows (input objects) with 8 variables: each integer valued variable was varied randomly between 1 and 100. Odd–Even data set Data Matrix (8  400) Class (1  400) if per row the total number of even values was greater than the total number of odd values, the class membership for that row was assigned to be 1, otherwise the class membership was set to −1

20. Output map by C CC CPNN for Odd-Even data Output layer Input layer Input Class CPNN  Scattered pattern of output map  No real relationship between the multivariate topological structure present in the input space and the associated class membership of the objects.  Does not take into account the class information during the formation of the input and output maps.

21. Output map by S SS SKNN for Odd-Even data Input ClassSKNN Kohonen layer  Scattered pattern of output map  Imbalance between the number of input (8) and output (1) variables  Does not take into account the class information during the formation of the input and output maps.

22. Output layer Output Similarity map Fused Similarity map Input Similarity map Input Class Input Kohonen layer X-Y Fused Network Output map by X XX XYFN for Odd-Even data  Nice coherent output maps  Indicating a certain ‘hidden’ relationship present between the input and output space.

23. Class 1 Class 2 Class 3 Set 1 X Y Z Class 1 Data Matrix (3  450) Set 2Set 3 Class 2 Class 1 1 0 0 0 1 0 0 0 1 Three normally distributed clouds of data points in three dimensions. the first 150 objects belong to a multivariate normal distribution around the origin (class 3), whereas the other two classes 1 and 2, each consisting of 150 objects as well, are normally distributed around the centroids (5,3,4) and (5.5, 3.5, 4.5). Overlap data set Input layer Output layer We will compares the quality of the 8×8 input and output maps for the CPN and XYF networks for the Overlap data set.

24. Xmap Ymap Class map CPNN Input layer Output layer

25. XY-Fused Network for Overlap data set Class map Ymap Xmap

26. Neural Networks For Chemists, An Introduction. (Weinheim/VCH Publishers ) Chem.Int.Lab.sys. 83 (2006) 99–113 Chem.Int.Lab.sys 90 (2008) 84–91 Chem.Int.Lab.sys. 38 (1997) 1-23 Current Computer-Aided Drug Design, 2005, 1, 73-78 References

27. Thanks

28. 0.2 0.4 0.1 0.4 0.5 0.1 0.3 0.6 0.8 0.0 0.7 0.2 0.9 0.2 0.4 0.3 0.1 0.8 0.9 0.2 0.4 0.5 0.1 0.5 0.0 0.6 0.3 0.7 0.0 0.1 0.2 0.9 0.1 1.0 0.0 0.1 0.2 0.3 0.8 0.7 0.4 0.7 0.2 0.7 4×4×2 1.0 0.2 0.6 Input vector output 0.340.800.520.76 1.280.460.801.18 0.820.300.760.44 1.060.321.181.16 Winner

29. 0.2 0.4 0.1 0.4 0.5 0.1 0.3 0.6 0.8 0.0 0.7 0.2 0.9 0.2 0.4 0.3 0.1 0.8 0.9 0.2 0.4 0.5 0.1 0.5 0.0 0.6 0.3 0.7 0.0 0.1 0.2 0.9 0.1 1.0 0.0 0.1 0.2 0.3 0.8 0.7 0.4 0.7 0.2 0.7 1.0 0.2 0.6 Input vector 0.8 -0.2 0.5 0.6 -0.3 0.1 0.9 -0.1 0.0 0.4 -0.6 0.6 0.3 0.0 -0.3 0.8 -0.2 0.3 0.7 0.1 -0.2 0.1 0.0 0.2 0.5 0.1 1.0 -0.4 0.3 0.2 0.5 0.8 -0.7 0.5 0.0 0.2 0.5 0.9 0.0 0.3 0.2 -0.5 0.2 0.3 0.0 -0.1 1× 0.9× 0.8×0.9× 0.6×0.9× × 0.4×0.9 a max =0.9 a min =0.1 t=1 (first epoch) Neighbor function: Linear winner  d