1. Migratory File Services for Batch-Pipelined Workloads
John Bent, Douglas Thain, Andrea Arpaci-Dusseau,
Remzi Arpaci-Dusseau, and Miron Livny
WiND and Condor Projects
6 May 2003
Hello...
Joint work between the WiND and Condor groups.
John just described batch-pipelined applications, a common application model that has a regular structure and significant I/O needs.
In this talk, I will describe migratory file services, a system model for executing batch-pipelined workloads.

2. How to Run a Batch-Pipelined Workload?
Shared
Data a1 a2 a3
pipe.1
pipe.2
pipe.3 b1 b2 b3
out.1
out.2
out.3

3. Cluster-to-Cluster Computing
Grid Engine Cluster
The
Internet
PBS Cluster
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Home
System
Condor Pool
Node
Node
Node
Node
Archive
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node

4. How to Run a Batch-Pipelined Workload?
“Remote I/O”
Submit jobs to a remote batch system.
Let all I/O come directly home.
Inefficient if re-use is common.
(( But perfect if no data sharing! ))
“FTP-Net”
User finds remote clusters.
Manually stages data in.
Submits jobs, deals with failures.
Pulls data out.
Lather, rinse, repeat.

5. Hawk: A Migratory File Service for Batch-Pipelined Workloads
Automatically deploys a “task force” across multiple wide-area systems.
Manages applications from a high level, using knowledge of process interactions.
Provides dependable performance with peer-to-peer techniques. (( Locality is key! ))
Understands and reacts to failures using knowledge of the system and workloads.

6. Dangers Failures Physical: Networks fail, disks crash, CPUs halt.
Logical: Out of space/memory, lease expired.
Administrative: You can’t use cluster X today.
Dependencies
A comes before C and D, which are simultaneous.
What do we do if the output of C is lost?
Risk vs Reward
A gamble: Staging input data to a remote CPU.
A gamble: Leaving output data at a remote CPU.
Dangers associated with distributed computing.
Remote I/O simplifies much of these dangers -- you never lose data remotely, but it is extraordinarily slow.

7. Hawk In Action Home System Batch-Pipelined Workload
Grid Engine Cluster
The
Internet
Hawk
PBS Cluster
Node
Node a2 b2 c2
Node o2 i2 a3 b3 c3
Node
Node
Node o3 i3
Node
Node
Node
Node
Node
Node
Node
Node
Node
Home
System
Condor Pool
Node
Node
Node a1 b1 c1
Node i1
Archive o1
Node
Node
Node
Node
Node
Node
Node
Node o1 o2 o3
Node
Node
Node
Node i1 i2 i3

8. Workflow Language 1 (Start With Condor DAGMan)
job a a.condor
job b b.condor
job c c.condor
job d d.condor
parent a child c
parent b child d a b c d
The key to Hawk is that it allows the user to specify a complex batch-parallel workload in an abstract manner. The workflow is then mapped into the system.
I’m going to begin by showing you the Hawk workflow language.

9. Workflow Language 2 a b c d v1 Archive Storage mydata volume v1
ftp://archive/mydata
mount v1 a /data
mount v1 b /data
volume v2 scratch
mount v2 a /tmp
mount v2 c /tmp
volume v3 scratch
mount v3 b /tmp
mount v3 d /tmp v2 v3 a b c d

10. Workflow Language 3 a b c d v1 Archive Storage mydata v2 v3
extract v2 x ftp://home/out.1
extract v3 x ftp://home/out.2 x x c d
out.1
out.2

11. Mapping the Workflow to the Migratory File System
Abstract Jobs
Become jobs in a batch system
May start, stop, fail, checkpoint, restart...
Logical “scratch” volumes
Become temporary containers on a scratch disk.
May be created, replicated, and destroyed...
Logical “read” volumes
Become blocks in a cooperative proxy cache.
May be created, cached, and evicted...

12. System Components Node Node Node Node Node Node Node Node Node Node
PBS Cluster
Condor Pool
Archive
Condor MM
Condor
SchedD
Workflow
Manager

13. Gliding In Node Node Node Node Node Node Node Node Node Node
StartD
Proxy
Master
Node
StartD
Proxy
Master
StartD
Proxy
Master
Node
Node
Node
StartD
Proxy
Master
Node
Node
Node
Node
StartD
Proxy
Master
PBS Cluster
Condor Pool
Archive
Condor MM
Condor
SchedD
StartD
Proxy
Master
Glide-In
Job

14. System Components Node Node Node Node Node Node Node Node Node Node
Proxy
Proxy
Proxy
StartD
StartD
StartD
Node
Node
Node
Node
Node
Proxy
Proxy
StartD
StartD
PBS Head Node
Condor Pool
Archive
Condor MM
Condor
SchedD
Workflow
Manager

15. Cooperative Proxies Node Node Node Node Node Node Node Node Node Node
Proxy
Proxy
Proxy
StartD
StartD
StartD
Node
Node
Node
Node
Node
Proxy
Proxy
StartD
StartD
PBS Head Node
Condor Pool
Archive
Condor MM
Condor
SchedD
Workflow
Manager

16. System Components Node Node Node Node Node Node Node Node Node Node
Proxy
Proxy
Proxy
StartD
StartD
StartD
Node
Node
Node
Node
Node
Proxy
Proxy
StartD
StartD
PBS Head Node
Condor Pool
Archive
Condor MM
Condor
SchedD
Workflow
Manager

17. Batch Execution System
Node
Node
Node
Node
Node
Proxy
Proxy
Proxy
StartD
Node
Node
Node
Node
Node
Proxy
Proxy
PBS Head Node
Condor Pool
Archive
Condor MM
Condor
SchedD
Workflow
Manager

18. System Components Node PBS Head Node Condor Pool Archive Workflow
Proxy
PBS Head Node
Condor Pool
StartD
Proxy
StartD
Archive
Condor MM
Condor
SchedD
Workflow
Manager

19. Workflow Manager Detail
Archive
Proxy
StartD
Condor MM
Condor
SchedD
Workflow
Manager

20. Proxy StartD Archive Archive Workflow Manager Condor MM Condor SchedD

21. Proxy StartD Coop Block Input Cache Create Container Archive Workflow
Local Area Network
Container 120
Create
Container
Wide Area Network
Archive
/mydata
d15
d16
Condor MM
Condor
SchedD
Workflow
Manager

22. POSIX Library Interface
Proxy
StartD
Job
Coop
Block
Input
Cache
open(“/data/d15”);
creat(“/tmp/outfile”);
Local Area Network
Container 120
POSIX Library Interface
Agent
/tmp cont://host5/120
/data cache://host5/archive/mydata
outfile
Wide Area Network
Archive
Execute
Job
/mydata
d15
d16
Condor MM
Condor
SchedD
Workflow
Manager

23. Proxy StartD Job Completed Coop Block Input Cache Extract Output
Local Area Network
Container 120
outfile
Extract
Output
Wide Area Network
Archive
/mydata
Condor MM
Condor
SchedD
Workflow
Manager
d15
out65
d16

24. Proxy StartD Job Completed Coop Block Input Cache Delete Container
Local Area Network
Container 120
outfile
Delete
Container
Wide Area Network
Archive
/mydata
Condor MM
Condor
SchedD
Workflow
Manager
d15
out65
d16

25. Proxy StartD Job Completed Coop Block Input Cache Container Deleted
Local Area Network
Container
Deleted
Wide Area Network
Archive
/mydata
Condor MM
Condor
SchedD
Workflow
Manager
d15
out65
d16

26. Fault Detection and Repair
The proxy, startd, and agent detect failures:
Job evicted by machine owner.
Network disconnection between job and proxy.
Container evicted by storage owner.
Out of space at proxy.
The workflow manager knows the consequences:
Job D couldn’t perform it’s I/O.
Check: Are volumes V1 and V3 still in place?
Aha: Volume V3 was lost -> Run B to create it.

27. Performance Testbed Controlled “remote” cluster:
32 cluster nodes at UW.
Hawk submitter also at UW.
Connected by a restricted 800 Kb/s link.
Also some preliminary tests on uncontrolled systems:
Hawk over PBS cluster at Los Alamos
Hawk over Condor system at INFN Italy.

28. Batch-Pipelined Applications
Name
Stages
Load
Remote (jobs/hr)
Hawk (jobs/hr)
BLAST 1
Batch
Heavy
4.67
747.40
CMS 2
and Pipe
33.78 HF 3
Pipe
40.96

29. Rollback
Cascading
Failure
Failure
Recovery

30. A Little Bit of Philosophy
Most systems build from the bottom up:
“This disk must have five nines, or else!”
MFS works from the top down:
“If this disk fails, we know what to do.”
By working from the top down, we finesse many of the hard problems in traditional filesystems.

31. Future Work Integration with Stork
P2P Aspects: Discovery & Replication
Optional Knowledge: Size & Time
Delegation and Disconnection
Names, names, names:
Hawk – A migratory file service.
Hawkeye – A system monitoring tool.

32. Let Hawk juggle your work!
jobs
Let Hawk juggle your work!
job
data
data ?
Feeling
overwhelmed?