Department of Computer Science University of the West Indies Part II

Parallel Programming? ENIAC, University of Pennsylvania 1946 (

The Need For Power

Computational Science qTraditional scientific and engineering paradigm q Do theory or paper design q Perform experiments or build system qReplacing both by numerical experiments q Real phenomena are too complicated to model by hand q Real experiments are: q too hard, e.g., build large wind tunnels q too expensive, e.g., build a throw-away passenger jet q too slow, e.g., wait for climate or galactic evolution q too dangerous, e.g., weapons, drug design

Computational Science Examples qAstrophysical thermonuclear flashes qNuclear weapons qWeather prediction qClimate and atmospheric modeling qDrug design qBlood flow qFluid dynamics (CFD)

Fluid Dynamics qForced convective heat transfer qBuoyant convection qHairpin vortex generation qRayleigh-Taylor instability

Hairpin Vortices - Transition to Turbulence qBoundary layer flow past a hemispherical roughness element qRe= based on hemisphere height qK= spectral elements of polynomial degree N=7-15

Simulation Cost qCost is O(Re 3 ) qRe=1K simulation ~ 1 week on 512 processors of ASCI Red q 50GF, 64 GB qRe=10K ~ 1 year on all 8192 processors of ASCI Red q 800GF, 1TB qWe’re really interested in Re=1M … qCan’t even think of doing the Re=1K problem on a uniprocessor machine let alone the 10K or 1M problems!

The Necessity of Parallel Computing

How fast can a serial computer be? qConsider the 1 Tflop sequential machine q data must travel some distance, r, to get from memory to CPU q to get 1 data element per cycle, this means times per second at the speed of light, c = 3e8 m/s q r < c/10 12 = 0.3 mm qNow put 1 TB of storage in a.3 mm 2 area q each word occupies about 3 Angstroms 2, the size of a small atom r =.3 mm 1 Tflop 1 TB sequential machine

Even if we could make it... q... it’d be too expensive qMarket forces are dictating use of COTS

The Solution ? qAdd more workers! qUse a collection of processors and memory modules to work together to solve our problems qSupercomputers, MPPs, Clusters, Beowulfs

Bad News

Still Lots of Work qDecide on and implement an interconnection network for the processors and memory modules qDesign and implement system software for the hardware qDevise algorithms and data structures for solving our problems qDivide the algorithms and data structures up into subproblems qIdentify the communication that will be needed between the subproblems qAssign subproblems to processors and memory modules

Modern Layered Framework CAD MultiprogrammingShared address Message passing Data parallel DatabaseScientific modeling Parallel applications Programming models Communication abstraction User/system boundary Compilation or library Operating systems support Communication hardware Physical communication medium Hardware/software boundary