Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences Department, UW-Madison Gathering at the Well: Creating Communities for Grid I/O

New framework needed › Remote I/O is possible anywhere › Build notion of locality into system? › What are possibilities?  Move job to data  Move data to job  Allow job to access data remotely › Need framework to expose these policies

Key elements › Storage appliance, interposition agents, schedulers and match-makers › Mechanism not policies › Policies are exposed to an upper layer  We will however demonstrate the strength of this mechanism

To infinity and beyond › Speedups of 2.5x possible when we are able to use locality intelligently › This will continue to be important  Data sets are getting larger and larger  There will always be bottlenecks

Outline › Motivation › Components › Expressing locality › Experiments › Conclusion

I/O communities › Mechanism which allow either  jobs to move to data, or  data to move to jobs, or  data to be accessed remotely › Framework to evaluate these policies

Grocers, butchers, cops › Members of an I/O community  Storage appliances  Interposition agents  Scheduling systems  Discovery systems  Match-makers  Collection of CPU’s

Storage appliances › Should run without special privilege  Flexible and easily deployable  Acceptable to nervous sys admins › Should allow multiple access modes  Low latency local accesses  High bandwidth remote puts and gets

NeST Common protocol layer GFTPChirpHTTPFTP Dispatcher Storage Manager Physical storage layer Multiple concurrencies Transfer Manager Control flow Data flow

Interposition agents › Thin software layer interposed between application and OS › Allow applications to transparently interact with storage appliances › Unmodified programs can run in grid environment

PFS: Pluggable File System

Scheduling systems and discovery › Top level scheduler needs ability to discover diverse resources › CPU discovery  Where can a job run? › Device discovery  Where is my local storage appliance? › Replica discovery  Where can I find my data?

Match-making › Match-making is the glue which brings discovery systems together › Allows participants to indirectly identify each other  i.e. can locate resources without explicitly naming them

Condor and ClassAds

Outline › Motivation › Components › Expressing locality › Experiments › Conclusion

I/O Communities UW INFN

Two I/O communities › INFN Condor pool  236 machines, about 30 available at any one time  Wide range of machines and networks spread across Italy  Storage appliance in Bologna 750 MIPS, 378 MB RAM

Two I/O communities › UW Condor pool  ~900 machines, 100 dedicated for us  Each is 600 MIPS, 512 MB RAM  Networked on 100 Mb/s switch  One was used as a storage appliance

Who Am I This Time? › We assumed the role of an Italian scientist › Database stored in Bologna › Need to run 300 instances of simulator

Hmmm…

Three way matching Machine NeST Job Ad Machine Ad Storage Ad match Refers to NearestStorage. Knows where NearestStorage is.

Two way ClassAds Type = “job” TargetType = “machine” Cmd = “sim.exe” Owner = “thain” Requirements = (OpSys==“linux”) Job ClassAd Type = “machine” TargetType = “job” OpSys = “linux” Requirements = (Owner==“thain”) Machine ClassAd

Three way ClassAds Type = “job” TargetType = “machine” Cmd = “sim.exe” Owner = “thain” Requirements = (OpSys==“linux”) && NearestStorage.HasCMSData Job ClassAd Type = “machine” TargetType = “job” OpSys = “linux” Requirements = (Owner==“thain”) NearestStorage = ( Name = “turkey”) && (Type==“Storage”) Machine ClassAd Type = “storage” Name = “turkey.cs.wisc.edu” HasCMSData = true CMSDataPath = /cmsdata” Storage ClassAd

Outline › Motivation › Components › Expressing locality › Experiments › Conclusion

BOOM!

CMS simulator sample run › Purposefully choose a run with high I/O to CPU ratio › Accesses about 20 MB of data from a 300 MB database › Writes about 1 MB of output › ~160 seconds execution time  on a 600 MIPS machine with local disk

Policy specification › Run only with locality  Requirements = (NearestStorage.HasCMSData) › Run in only one particular community  Requirements = (NearestStorage.Name == “nestore.bologna”) › Prefer home community first  Requirements = (NearestStorage.HasCMSData)  Rank = (NearestStorage.Name == “nestore.bologna” ) ? 10 : 0 › Arbitrarily complex  Requirements = ( NearestStorage.Name == “nestore.bologna”) || ( ClockHour 18 )

Policies evaluated › INFN local › UW remote › UW stage first › UW local (pre-staged) › INFN local, UW remote › INFN local, UW stage › INFN local, UW local

Completion Time

CPU Efficiency

Conclusions › I/O communities expose locality policies › Users can increase throughput › Owners can maximize resource utilization

Future work › Automation  Configuration of communities  Dynamically adjust size as load dictates › Automation  Selection of movement policy › Automation

For more info › Condor  › ClassAds  › PFS  › NeST 

Local only

Remote only

Both local and remote

I/O communities are an old idea, right? › File servers and administrative domains › No, not really. We need  more flexible boundaries  simple mechanism by which users can express I/O community relationships  hooks into system that allow users to use locality

Grid applications have demanding I/O needs › Petabytes of data in tape repositories › Scheduling systems have demonstrated that there are idle CPUs › Some systems  move jobs to data  move data to jobs  allow job remote access to data › No one approach is always “best”

Easy come, easy go › In a computation grid, resources are very dynamic › Programs need rich methods for finding and claiming resources  CPU discovery  Device discovery  Replica discovery

Bringing it all together CPU Discovery System Replica Discovery System Device Discovery System JobAgent Execution site Storage appliance Distributed Repository Short-haul I/O Long-haul I/O

Conclusions › Locality is good › Balance point between staging data and accessing it remotely is not static  depends on specific attributes of the job data size, expected degree of re-reference, etc  depends on performance metric CPU efficiency or job completion time

Implementation › NeST  storage appliance › Pluggable File System (PFS)  interposition agent built with Bypass › Condor and ClassAds  scheduling system  discovery system  match-maker

Jim Gast and Bart say... › Too many bullet slides › Contributions  scientist doesn’t want to name bec resources are dynamic and name is irrelevant  hooks into system to allow users to express and take advantage of locality

Jim Gast and Bart say...  everyone knows locality is good - but there is no way to express this and run jobs on the grid  I/O communities are mechanism by which user can use locality and specify policies to optimize job performance

4 earth-shattering revelations 1) The grid is big. 2) Scientific data-sets are large. 3) Idle resources are available. 4) Locality is good.

Mechanisms not policies › I/O communities are a mechanism not a policy › A higher layer is expected to choose application appropriate policies › We will however demonstrate the strength of the mechanism by defining appropriate policies for one particular application

Experimental results › Implementation › Environment › Application › Measurements › Evaluation