VIPIN VIJAYAN 11/11/03 A Performance Analysis of Two Distributed Computing Abstractions

Outline Introduction Problem AllPairs Work Queue & Condor Evaluation Preliminary Results

Introduction In a biometric matching experiment, a query image is compared against a target image to compute a match score. F is the matching function. M ij = F(A i,B j ) BiBi AiAi F(A 1,B 1 ) Similarity matrix, M

Problem Problems encountered while computing the similarity matrix: O(n 2 ) increase in computational time where n is the number of images. Increase in computational time due to the size of the images and complexity of the matching function. Proposed solution:  a distributed computing abstraction such as AllPairs.

AllPairs: a distributed computing abstraction As an abstraction:  It simplifies the calculation of a similarity matrix:  We simply have to provide the matching function, and the two sets of images. Because of the abstraction:  We can take advantage of the particular abstraction to speed up computation.  Ex: Taking advantage of sub-matrices.

AllPairs: a second look Worker 1 (Busy) Worker 2 (Busy) Worker 3 (Busy) Worker 4 (Busy) Waiting Active Here we upload 4+3 = 7 files to a single worker. As opposed to uploading 4*3 = 12 files to different workers. So the main advantage of splitting it up into sub-matrices is decrease in uploaded data.

Work Queue & Condor Condor: A framework which can be used to manage workload on a cluster of computers.  It tries to make use of cycles from idle computers.  Any Condor process can be stopped by the user of the machine. Work Queue: A system which manages Condor, which works with machines of different processing power and scheduling policies and is fault-tolerant.  A stopped process, for example, is restarted automatically by Work Queue.

Evaluation Metrics Speed-up:  Ratio of execution time using 1 processor to execution time using p processors. Scale-up:  Ratio to execution time using 1 processor to execution time using p processor where problem size is scaled by p.  Here we define problem size as the number of images we use. Overhead:  Overall time subtracted by core computation time.  This will mainly be from I/O operations and communication time between master and workers.

Speed-up

Scale-up

Overhead w.r.t problem size

Overhead w.r.t number of workers

Questions?