Chen Jin (HT016952H) Zhao Xu Ying (HT016907B) A Programming Model of Hybrid Distributed/Shared Memory System in Data Mining Field Chen Jin (HT016952H) Zhao Xu Ying (HT016907B) 9/12/2018

Outlines Introduction Mixed-mode Parallel Programming Fuzzy c-Medoids Algorithm (FCMdd) Serial Version of FCMdd MPI Version of FCMdd Mixed-mode Version of FCMdd Comparisons between Pure MPI & Mix-mode Result Analysis Conclusion

Introduction This report discusses the benefits of developing mixed-mode MPI/OpenMP applications on clustered SMPs in data mining field, especially focusing on web log clustering. We start with a realistic serial C++ FCMdd program. Next we show some of the modifications that were performed to the program to enable it to run using MPI

Introduction (Contd) Then we show the simple modifications that were performed to the source to take advantage of OpenMP. We show that using a combination of MPI and OpenMP can be an effective method of programming hybrid systems.

Mixed-mode Parallel Programming Shared-memory architectures are gradually becoming more prominent Advances in technology have allowed larger numbers of CPUs to have access to a single memory space But it is not immediately clear that message passing is the most efficient parallelization technique within an SMP box. Message-passing codes written in MPI are obviously portable and should transfer easily to clustered SMP systems. In theory a shared memory model such as OpenMP should be preferable.

Mixed-mode Cluster Hierarchical Model

Mixed-mode Parallel Programming (Contd) Hypothesis: A combination of shared-memory and message-passing parallelization paradigms within the same application may provide a more efficient parallelization strategy than pure MPI.

FCMdd Algorithm Description Web mining can be viewed as the extraction of structure from unlabeled semistructured data containing information. Three operations of particular interest: Clustering – finding natural groupings of users, pages. Associations – URLs tend to be requested together. Sequential analysis – URLs tend to be accessed.

Fuzzy c-Medoids Algorithm(FCMdd) Dissimilarity: Objective function of FCMdd (1) (2)

The Role of FUZZY Granularity in Web Mining The categories and associations in Web mining do not have crisp boundaries. They overlap considerably and are best described by fuzzy sets. Bad exemplars (outliers) and incomplete data can easily occur in the data set

Serial Version of FCMdd Fix the number of clusters c: Set iter = 0; Pick initial medoids from Repeat Compute memberships for and by using (2) and identify (A) Store the current medoids: Compute the new medoids for (B) Until ( or )

Properties of FCMdd Dealing with data which size is extremely large Time complexity is O(n*c*p), where n is the data size

MPI Version of FCMdd Read Data from Disk to File Different node reads different parts of file, then broadcast to others. Different node reads the same file separately. A specified node reads all the data then broadcasts to the other nodes

MPI Version of FCMdd(Contd) Choose Medoids from sessions The kernel of FCMdd is the replacement of original medoid by appropriate candidates. Medoids can be considered separately in each iteration. Carbine the medoids parts together to get the complete array.

MPI Version of FCMdd(Contd)

Performance Analysis of MPI mode Record number is 138,384: 1 node : 15.1244s 2 nodes : 7.7100s 4 nodes : 6.3486s 8 nodes : 5.1482s

Performance Analysis of MPI mode(Contd) Record Number is 1,882,384: 1 node : 68.3798s 2 nodes : 33.6380s 4 nodes : 26.1140s 8 nodes : 21.5922s

Mix-mode version of FCMdd Two kinds of Mix-mode programming Models: MPI parallelization occurs across hosts at the top level, and OpenMP parallelization occurring below within each node. Environment variable OMP_NUM_THREADS can be set for each host, which may be different or equivalent to set differently for each host. MPI and OpenMP parallelization occurs within a host. Environment variable can be used to set the number of OpenMP threads for all MPI processes and all MPI processes will use the same number of threads. If each MPI process needs different number of threads, omp_set_num_threads should be called on each MPI process.

Mix-mode version of FCMdd(Contd) The left figure corresponds to the parallelization for a cluster of uniprocessor nodes, where a MPI process is allocated on each node.

Mix-mode version of FCMdd(Contd) The right figure shows how the computation part of the process is split into threads by using OpenMP directives. P is the part of code that can not be parallelized with OpenMP and Pn is the OpenMP parallel part.

Mix-mode version of FCMdd(Contd)

Mix-mode version of FCMdd(Contd)

Comparisons Between the MPI and Mix-mode Programming we tested on hydra within one node by varied database size

Comparisons Between the MPI and Mix-mode Programming

Comparisons Between the MPI and Mix-mode Programming Data size is 610K in Beowulf:

Comparisons Between the MPI and Mix-mode Programming

Comparisons Between the MPI and Mix-mode Programming Data size is 8M on Beowulf:

Comparisons Between the MPI and Mix-mode Programming

Comparisons Between the MPI and Mix-mode Programming (Contd) The two modes have similar performance. However, there are still differences.

Comparisons Between the MPI and Mix-mode Programming (Contd) Within one host, Mix-mode shows more potential to improve efficiency than pure MPI programming. Our test result in hydra shows that inside one host, Mix-mode is better than pure MPI mode. In theory, OpenMP makes better usage of shared memory architecture in pure shared memory mechanism.

Comparisons Between the MPI and Mix-mode Programming (Contd) In a hybrid shared/distributed memory system, a mix mode implementation could be more efficient for large problems giving poor scalability for a large number of processors.

Comparisons Between the MPI and Mix-mode Programming (Contd) In theory, a mixed mode code, with the MPI parallelization occurring across the SMP hosts and OpenMP parallelization within the hosts, should be more efficient on an SMPs cluster as the model matches the architecture more closely than a pure MPI model. The test results in beowulf also support our conclusion.

Conclusion Hypothesis: A combination of shared-memory and message-passing parallelization paradigms within the same application may provide a more efficient parallelization strategy than pure MPI. Our project supports the hypothesis successfully.

Contribution Chen Jin Zhao XuYing Data Preprocessing Serial version MPI version MIX version Comparison

Reference [1] Franck Cappello and Daniel Etiemble. MPI versus MPI+OpenMP on the IBM SP for the NAS Benchmarks. In 0-7803-9802-5/2000 (c) 2000 IEEE. [2] L.A.Smith. Mixed Mode MPI/OpenMP Programming. In Edinburgh Parallel Computing Centre, Edinburgh, EH93 JZ [3] Franck Cappello, Olivier Richard and Daniel Etiemble. Investigating the performance of two programming models for clusters of SMPPCs. In High-Performance Computer Architecture, 2000. HPCA-6. Proceedings. Sixth International Symposium on , 1999 [4] Piero Lanucara and Sergio Rovida. Conjugate-Gradients Algorithms: An MPI-OpenMP Implementation on. In CASPUR, c/o Universit`a“LaSapienza”, P. Aldo Moro 2,00185 Roma, Italy and Istituto Analisi Numerica-C.N.R., via Ferrata1, 27100 Pavia, Italy [5] OpenMP, OpenMP C and C++ Application Program Interface. Http: // www.openmp.org/. [6] North Carolina Supercomputing Center. A Standard Shared Memory Parallel Computing. http://www.ncsc.org/training/materials/openmp/sginotes/sld001.htm

END