Migratory File Services for Batch-Pipelined Workloads John Bent, Douglas Thain, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, and Miron Livny WiND and Condor Projects 6 May 2003 Hello... Joint work between the WiND and Condor groups. John just described batch-pipelined applications, a common application model that has a regular structure and significant I/O needs. In this talk, I will describe migratory file services, a system model for executing batch-pipelined workloads.

How to Run a Batch-Pipelined Workload? Shared Data a1 a2 a3 pipe.1 pipe.2 pipe.3 b1 b2 b3 out.1 out.2 out.3

Cluster-to-Cluster Computing Grid Engine Cluster The Internet PBS Cluster Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Home System Condor Pool Node Node Node Node Archive Node Node Node Node Node Node Node Node Node Node Node Node

How to Run a Batch-Pipelined Workload? “Remote I/O” Submit jobs to a remote batch system. Let all I/O come directly home. Inefficient if re-use is common. (( But perfect if no data sharing! )) “FTP-Net” User finds remote clusters. Manually stages data in. Submits jobs, deals with failures. Pulls data out. Lather, rinse, repeat.

Hawk: A Migratory File Service for Batch-Pipelined Workloads Automatically deploys a “task force” across multiple wide-area systems. Manages applications from a high level, using knowledge of process interactions. Provides dependable performance with peer-to-peer techniques. (( Locality is key! )) Understands and reacts to failures using knowledge of the system and workloads.

Dangers Failures Physical: Networks fail, disks crash, CPUs halt. Logical: Out of space/memory, lease expired. Administrative: You can’t use cluster X today. Dependencies A comes before C and D, which are simultaneous. What do we do if the output of C is lost? Risk vs Reward A gamble: Staging input data to a remote CPU. A gamble: Leaving output data at a remote CPU. Dangers associated with distributed computing. Remote I/O simplifies much of these dangers -- you never lose data remotely, but it is extraordinarily slow.

Hawk In Action Home System Batch-Pipelined Workload Grid Engine Cluster The Internet Hawk PBS Cluster Node Node a2 b2 c2 Node o2 i2 a3 b3 c3 Node Node Node o3 i3 Node Node Node Node Node Node Node Node Node Home System Condor Pool Node Node Node a1 b1 c1 Node i1 Archive o1 Node Node Node Node Node Node Node Node o1 o2 o3 Node Node Node Node i1 i2 i3

Workflow Language 1 (Start With Condor DAGMan) job a a.condor job b b.condor job c c.condor job d d.condor parent a child c parent b child d a b c d The key to Hawk is that it allows the user to specify a complex batch-parallel workload in an abstract manner. The workflow is then mapped into the system. I’m going to begin by showing you the Hawk workflow language.

Workflow Language 2 a b c d v1 Archive Storage mydata volume v1 ftp://archive/mydata mount v1 a /data mount v1 b /data volume v2 scratch mount v2 a /tmp mount v2 c /tmp volume v3 scratch mount v3 b /tmp mount v3 d /tmp v2 v3 a b c d

Workflow Language 3 a b c d v1 Archive Storage mydata v2 v3 extract v2 x ftp://home/out.1 extract v3 x ftp://home/out.2 x x c d out.1 out.2

Mapping the Workflow to the Migratory File System Abstract Jobs Become jobs in a batch system May start, stop, fail, checkpoint, restart... Logical “scratch” volumes Become temporary containers on a scratch disk. May be created, replicated, and destroyed... Logical “read” volumes Become blocks in a cooperative proxy cache. May be created, cached, and evicted...

System Components Node Node Node Node Node Node Node Node Node Node PBS Cluster Condor Pool Archive Condor MM Condor SchedD Workflow Manager

Gliding In Node Node Node Node Node Node Node Node Node Node StartD Proxy Master Node StartD Proxy Master StartD Proxy Master Node Node Node StartD Proxy Master Node Node Node Node StartD Proxy Master PBS Cluster Condor Pool Archive Condor MM Condor SchedD StartD Proxy Master Glide-In Job

System Components Node Node Node Node Node Node Node Node Node Node Proxy Proxy Proxy StartD StartD StartD Node Node Node Node Node Proxy Proxy StartD StartD PBS Head Node Condor Pool Archive Condor MM Condor SchedD Workflow Manager

Cooperative Proxies Node Node Node Node Node Node Node Node Node Node Proxy Proxy Proxy StartD StartD StartD Node Node Node Node Node Proxy Proxy StartD StartD PBS Head Node Condor Pool Archive Condor MM Condor SchedD Workflow Manager

System Components Node Node Node Node Node Node Node Node Node Node Proxy Proxy Proxy StartD StartD StartD Node Node Node Node Node Proxy Proxy StartD StartD PBS Head Node Condor Pool Archive Condor MM Condor SchedD Workflow Manager

Batch Execution System Node Node Node Node Node Proxy Proxy Proxy StartD Node Node Node Node Node Proxy Proxy PBS Head Node Condor Pool Archive Condor MM Condor SchedD Workflow Manager

System Components Node PBS Head Node Condor Pool Archive Workflow Proxy PBS Head Node Condor Pool StartD Proxy StartD Archive Condor MM Condor SchedD Workflow Manager

Workflow Manager Detail Archive Proxy StartD Condor MM Condor SchedD Workflow Manager

Proxy StartD Archive Archive Workflow Manager Condor MM Condor SchedD

Proxy StartD Coop Block Input Cache Create Container Archive Workflow Local Area Network Container 120 Create Container Wide Area Network Archive /mydata d15 d16 Condor MM Condor SchedD Workflow Manager

POSIX Library Interface Proxy StartD Job Coop Block Input Cache open(“/data/d15”); creat(“/tmp/outfile”); Local Area Network Container 120 POSIX Library Interface Agent /tmp cont://host5/120 /data cache://host5/archive/mydata outfile Wide Area Network Archive Execute Job /mydata d15 d16 Condor MM Condor SchedD Workflow Manager

Proxy StartD Job Completed Coop Block Input Cache Extract Output Local Area Network Container 120 outfile Extract Output Wide Area Network Archive /mydata Condor MM Condor SchedD Workflow Manager d15 out65 d16

Proxy StartD Job Completed Coop Block Input Cache Delete Container Local Area Network Container 120 outfile Delete Container Wide Area Network Archive /mydata Condor MM Condor SchedD Workflow Manager d15 out65 d16

Proxy StartD Job Completed Coop Block Input Cache Container Deleted Local Area Network Container Deleted Wide Area Network Archive /mydata Condor MM Condor SchedD Workflow Manager d15 out65 d16

Fault Detection and Repair The proxy, startd, and agent detect failures: Job evicted by machine owner. Network disconnection between job and proxy. Container evicted by storage owner. Out of space at proxy. The workflow manager knows the consequences: Job D couldn’t perform it’s I/O. Check: Are volumes V1 and V3 still in place? Aha: Volume V3 was lost -> Run B to create it.

Performance Testbed Controlled “remote” cluster: 32 cluster nodes at UW. Hawk submitter also at UW. Connected by a restricted 800 Kb/s link. Also some preliminary tests on uncontrolled systems: Hawk over PBS cluster at Los Alamos Hawk over Condor system at INFN Italy.

Batch-Pipelined Applications Name Stages Load Remote (jobs/hr) Hawk (jobs/hr) BLAST 1 Batch Heavy 4.67 747.40 CMS 2 and Pipe 33.78 1273.96 HF 3 Pipe 40.96 3187.22

Rollback Cascading Failure Failure Recovery

A Little Bit of Philosophy Most systems build from the bottom up: “This disk must have five nines, or else!” MFS works from the top down: “If this disk fails, we know what to do.” By working from the top down, we finesse many of the hard problems in traditional filesystems.

Future Work Integration with Stork P2P Aspects: Discovery & Replication Optional Knowledge: Size & Time Delegation and Disconnection Names, names, names: Hawk – A migratory file service. Hawkeye – A system monitoring tool.

Let Hawk juggle your work! jobs Let Hawk juggle your work! job data data ? Feeling overwhelmed?