Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences Department, UW-Madison Gathering at the Well: Creating Communities for Grid I/O

4 earth-shattering revelations 1) The grid is big. 2) Scientific data-sets are large. 3) Idle resources are available. 4) Locality is good.

How to optimize job placement on the grid? › Move data to the jobs. › Move jobs to the data. › Allow jobs to access data remotely. › Need framework for evaluation.  I/O communities

I/O Communities UW INFN

I/O communities are an old idea › File servers and administrative domains › But, we want  more flexible boundaries  simple mechanism by which users can express I/O community relationships

I/O communities › Mechanism which allow either  jobs to move to data, or  data to move to jobs, or  data to be accessed remotely › Framework to evaluate these policies

Grocers, butchers, cops › Members of an I/O community  Storage appliances  Interposition agents  Scheduling systems  Discovery systems  Match-makers  Collection of CPU’s

Storage appliances › Should run without special privilege  Flexible and easily deployable  Acceptable to nervous sys admins › Should allow multiple access modes  Low latency local accesses  High bandwidth remote puts and gets

Interposition agents › Thin software layer interposed between application and OS › Allow applications to transparently interact with storage appliances › Unmodified programs can run in grid environment

Scheduling systems and discovery › Top level scheduler needs ability to discover diverse resources › CPU discovery  Where can a job run? › Device discovery  Where is my local storage appliance? › Replica discovery  Where can I find my data?

Match-making › Match-making is the glue which brings discovery systems together › Allows participants to indirectly identify each other  i.e. can locate resources without explicitly naming them

Mechanisms not policies › I/O communities are a mechanism not a policy › A higher layer is expected to choose application appropriate policies › We will however demonstrate the strength of the mechanism by defining appropriate policies for one particular application

Experimental results › Implementation › Environment › Application › Measurements › Evaluation

Implementation › NeST  storage appliance › Pluggable File System (PFS)  interposition agent built with Bypass › Condor and ClassAds  scheduling system  discovery system  match-maker

Two I/O communities › INFN Condor pool  236 machines, about 30 available at any one time  Wide range of machines and networks spread across Italy  Storage appliance in Bologna 750 MIPS, 378 MB RAM

Two I/O communities › UW Condor pool  911 machines, 100 dedicated for us  Each is 600 MIPS, 512 MB RAM  Networked on 100 Mb/s switch  One was used as a storage appliance

CMS simulator sample run › Purposefully choose a run with high I/O to CPU ratio › Accesses about 20 MB of data from a 300 MB database › Writes about 1 MB of output › ~160 seconds execution time  on a 600 MIPS machine with local disk

Assume the position › We assumed the role of an Italian scientist › Database stored in Bologna › Need to run 300 instances of simulator › How to take advantage of UW pool?  Three way matching

Three way matching Machine NeST Job Ad Machine Ad Storage Ad match Refers to NearestStorage. Knows where NearestStorage is.

Two way ClassAds Type = “job” TargetType = “machine” Cmd = “sim.exe” Owner = “thain” Requirements = (OpSys==“linux”) Job ClassAd Type = “machine” TargetType = “job” OpSys = “linux” Requirements = (Owner==“thain”) Machine ClassAd

Three way ClassAds Type = “job” TargetType = “machine” Cmd = “sim.exe” Owner = “thain” Requirements = (OpSys==“linux”) && NearestStorage.HasCMSData Job ClassAd Type = “machine” TargetType = “job” OpSys = “linux” Requirements = (Owner==“thain”) NearestStorage = ( Name = “turkey”) && (Type==“Storage”) Machine ClassAd Type = “storage” Name = “turkey.cs.wisc.edu” HasCMSData = true CMSDataPath = /cmsdata” Storage ClassAd

Policy specification › Run anywhere where data is available  Requirements = (NearestStorage.HasCMSData) › Run local only  Requirements = (NearestStorage.Name == “nestore.bologna”) › Run local first  Requirements = (NearestStorage.HasCMSData)  Rank = (NearestStorage.Name == “nestore.bologna” ) ? 10 : 0 › Arbitrarily complex  Requirements = ( NearestStorage.Name == “nestore.bologna”) || ( ClockHour 18 )

Policies evaluated › INFN local › UW remote › UW stage first › UW local (pre-staged) › INFN local, UW remote › INFN local, UW stage › INFN local, UW local

Completion Time

CPU Efficiency

Conclusions › Locality is good › I/O communities are a natural structure to expose this locality › Users can use I/O communities to easily express different job placement policies

Future work › Automation  Configuration of communities  Dynamically adjust size as load dictates › Automation  Selection of movement policy › Automation

For more info › Condor  › ClassAds  › PFS  › NeST 

Local only

Remote only

Both local and remote

Grid applications have demanding I/O needs › Petabytes of data in tape repositories › Scheduling systems have demonstrated that there are idle CPUs › Some systems  move jobs to data  move data to jobs  allow job remote access to data › No one approach is always “best”

Easy come, easy go › In a computation grid, resources are very dynamic › Programs need rich methods for finding and claiming resources  CPU discovery  Device discovery  Replica discovery

Bringing it all together CPU Discovery System Replica Discovery System Device Discovery System JobAgent Execution site Storage appliance Distributed Repository Short-haul I/O Long-haul I/O

Conclusions › Locality is good › Balance point between staging data and accessing it remotely is not static  depends on specific attributes of the job data size, expected degree of re-reference, etc  depends on performance metric CPU efficiency or job completion time