May 19 Lecture Outline Introduce MPI functionality Introduce Doc Classification problem Parallel algorithm design Creating communicators Non-blocking communications Implementation Pipelining Clustering

Implementation of a Very Simple Document Classifier Manager/Worker Design Strategy Manager description (create initial tasks and communicate to/from workers) Worker description (receive tasks, enter an alternating communication from/to master and computation

Structure of Main program: Manager/Worker Paradigm MPI_Init (&argc, &argv); // what is my rank? MPI_Comm_rank(MPI_COMM_WORLD, &myrank); // how many processors are there? MPI_Comm_size(MPI_COMM_WORLD, &p); if (myid == 0) Manager(p); else Worker(myid, p); MPI_Barrier(MPI_COMM_WORLD); MPI_Finalize(); return(0);

Creating a Workers-only Communicator To support workers-only broadcast, need workers-only communicator Can use MPI_Comm_split Excluded processes (e.g., Manager) passes MPI_UNDEFINED as the value of split_key, meaning it will not be part of any new communicator

Workers-only Communicator int id; MPI_Comm worker_comm; ... if (!id) /* Manager */ MPI_Comm_split (MPI_COMM_WORLD, MPI_UNDEFINED, id, &worker_comm); else /* Worker */ MPI_Comm_split (MPI_COMM_WORLD, 0, id, &worker_comm);

Nonblocking Send / Receive MPI_Isend, MPI_Irecv initiate operation MPI_Wait blocks until operation complete Calls can be made early MPI_Isend as soon as value(s) assigned MPI_Irecv as soon as buffer available Can eliminate a message copying step Allows communication / computation overlap

Receiving Problem Worker does not know length of message it will receive Example, the length of File Path Name Alternatives Allocate huge buffer Check length of incoming message, then allocate buffer We’ll take the second alternative

Function MPI_Probe int MPI_Probe ( int src, int tag, MPI_Comm comm, MPI_Status *status ) Blocks until message is available to be received from process with rank src with message tag tag; status pointer gives info on message size.

Function MPI_Get_count int MPI_Get_count ( MPI_Status *status, MPI_Datatype dtype, int *cnt ) cnt returns the number of elements in message

MPI_Testsome Often need to check whether one or more messages have arrived Manager posts a nonblocking receive to each worker process Builds an array of handles or request objects Testsome allows manager to determine how many messages have arrived

Document Classification Problem Search directories, subdirectories for documents (look for .html, .txt, .tex, etc.) Using a dictionary of key words, create a profile vector for each document Store profile vectors

Data Dependence Graph (1)

Partitioning and Communication Most time spent reading documents and generating profile vectors Create two primitive tasks for each document

Data Dependence Graph (2)

Agglomeration and Mapping Number of tasks not known at compile time Tasks do not communicate with each other Time needed to perform tasks varies widely Strategy: map tasks to processes at run time

Manager/worker-style Algorithm Task/Functional Partitioning Domain/Data Partitioning

Roles of Manager and Workers

Manager Pseudocode Identify documents Receive dictionary size from worker 0 Allocate matrix to store document vectors repeat Receive message from worker if message contains document vector Store document vector endif if documents remain then Send worker file name else Send worker termination message until all workers terminated Write document vectors to file

Worker Pseudocode Send first request for work to manager if worker 0 then Read dictionary from file endif Broadcast dictionary among workers Build hash table from dictionary Send dictionary size to manager repeat Receive file name from manager if file name is NULL then terminate endif Read document, generate document vector Send document vector to manager forever

Task/Channel Graph

Enhancements Finding middle ground between pre-allocation and one-at-a-time allocation of file paths Pipelining of document processing

Allocation Alternatives n/p Load imbalance Time Excessive communication overhead 1 Documents Allocated per Request

Pipelining

Time Savings through Pipelining

Summary Manager/worker paradigm New tools for “kit” Dynamic number of tasks Variable task lengths No communications between tasks New tools for “kit” Create manager/worker program Create workers-only communicator Non-blocking send/receive Testing for completed communications Next Step: Cluster Profile Vectors

K-Means Clustering Assumes documents are real-valued vectors. Assumes distance function on vector pairs Clusters based on centroids (aka the center of gravity or mean) of points in a cluster, c: Reassignment of instances to clusters is based on distance of vector to the current cluster centroids. (Or one can equivalently phrase it in terms of similarities)

K-Means Algorithm Let d be the distance measure between instances. Select k random instances {s1, s2,… sk} as seeds. Until clustering converges or other stopping criterion: For each instance xi: Assign xi to the cluster cj such that d(xi, sj) is minimal. // Now Update the seeds to the centroid of each cluster) For each cluster cj sj = (cj)

K Means Example (K=2) Pick seeds Reassign clusters Compute centroids Converged!

Termination conditions Desire that docs in a cluster are unchanged Several possibilities, e.g., A fixed number of iterations. Doc partition unchanged. Centroid positions don’t change. We’ll choose termination when only small fraction change (threshold value)

Quiz: Describe Manager/Worker pseudo-code that implements the K-means algorithm in parallel What data partitioning for parallelism? How are cluster centers updated and distributed?

Hints objects to be clustered are evenly partitioned among all processes cluster centers are replicated Global-sum reduction on cluster centers is performed at the end of each iteration to generate the new cluster centers. Use MPI_Bcast and MPI_Allreduce