1. Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison tannenba@cs.wisc.edu http://www.cs.wisc.edu/condor Condor and Grid Challenges HEPiX Spring 2005 Karlsruhe, Germany

2. www.cs.wisc.edu/condor 2 Outline › Introduction to the Condor Project › Introduction to Condor / Condor-G › The Challenge: Take Grid Planning to the next level  How to address mismatch between local –vs- grid scheduler capabilities › Lessons learned

3. www.cs.wisc.edu/condor 3 The Condor Project (Established ‘85) Distributed High Throughput Computing research performed by a team of ~35 faculty, full time staff and students.

4. www.cs.wisc.edu/condor 4 The Condor Project (Established ‘85) Distributed High Throughput Computing research performed by a team of ~35 faculty, full time staff and students who:  face software engineering challenges in a distributed UNIX/Linux/NT environment  are involved in national and international grid collaborations,  actively interact with academic and commercial users,  maintain and support large distributed production environments,  and educate and train students. Funding – US Govt. (DoD, DoE, NASA, NSF, NIH), AT&T, IBM, INTEL, Microsoft, UW-Madison, …

5. www.cs.wisc.edu/condor 5 A Multifaceted Project › Harnessing the power of clusters – dedicated and/or opportunistic (Condor) › Job management services for Grid applications (Condor-G, Stork) › Fabric management services for Grid resources (Condor, GlideIns, NeST) › Distributed I/O technology (Parrot, Kangaroo, NeST) › Job-flow management (DAGMan, Condor, Hawk) › Distributed monitoring and management (HawkEye) › Technology for Distributed Systems (ClassAD, MW) › Packaging and Integration (NMI, VDT)

6. www.cs.wisc.edu/condor 6 Some free software produced by the Condor Project › Condor System › ClassAd Library › DAGMan › Fault Tolerant Shell (FTSH) › Hawkeye › GCB › MW › NeST › Stork › Parrot › VDT › And others… all as open source Data!

7. www.cs.wisc.edu/condor 7 Full featured system › Flexible scheduling policy engine via ClassAds  Preemption, suspension, requirements, preferences, groups, quotas, settable fair-share, system hold… › Facilities to manage BOTH dedicated CPUs (clusters) and non-dedicated resources (desktops) › Transparent Checkpoint/Migration for many types of serial jobs › No shared file-system required › Workflow management (inter-dependencies) › Support for many job types – serial, parallel, etc. › Fault-tolerant: can survive crashes, network outages, no single point of failure. › API: via SOAP / web services, DRMAA (C), Perl package, GAHP, flexible command-line tools › Platforms: Linux i386/IA64, Windows 2k/XP, MacOS, Solaris, IRIX, HP-UX, Compaq Tru64, … lots.

8. www.cs.wisc.edu/condor 8 Who uses Condor? › Commercial  Oracle, Micron, Hartford Life Insurance, CORE, Xerox, Exxon/Mobil, Shell, Alterra, Texas Instruments, … › Research Community  Universities, Gov Labs  Bundles: NMI, VDT  Grid Communities: EGEE/LCG/gLite, Particle Physics Data Grid (PPDG), CDF @ Fermi, USCMS, LIGO, iVDGL, NSF Middleware Initiative GRIDS Center, …

9. www.cs.wisc.edu/condor 9 Condor in a nutshell › Condor is a set of distributed services  A scheduler service (schedd) job policy  A resource manager service (startd) resource policy  A matchmaker service administrator policy › These services use ClassAds as the lingua franca

10. www.cs.wisc.edu/condor 10 ClassAds  are a set of uniquely named expressions; each expression is called an attribute and is an attribute name/value pair  combine query and data  extensible  semi-structured : no fixed schema (flexibility in an environment consisting of distributed administrative domains)  Designed for “MatchMaking”

11. www.cs.wisc.edu/condor 11 Arch = “Intel-Linux”; KFlops = 2200 Memory = 1024 Disk = 10000 Requirements = other.ImageSize < Memory preferred_projects = { “atlas", “uscms" } Rank = member(other.project, preferred_projects) * 500 … (70+ out of the box, can be customized) Sample – Machine ClassAd

12. www.cs.wisc.edu/condor 12 Another Sample - Job Submit project = “uscms” imagesize = 260 requirements = (other.Arch == “Intel-Linux” || other.Arch == “Sparc-Solaris”) && Disk > 500 && ( fooapp_path =!= UNDEFINED ) rank = other.KFlops * 1000 + other.Memory executable = find_particle.$$(Arch) arguments = $$(fooapp_path) … Note: We introduced augmentation of the job ClassAd based upon information discovered in its matching resource ClassAd.

13. www.cs.wisc.edu/condor 13 First make some matches Schedd Startd Schedd MatchMaker Jobs NOTE: Have as many schedds as you want!

14. www.cs.wisc.edu/condor 14 Then claim and schedule Schedd Startd Schedd MatchMaker Jobs

15. www.cs.wisc.edu/condor 15 Then claim and schedule Schedd Startd Schedd MatchMaker Jobs Note: Each schedd decides for itself which job to run.

16. www.cs.wisc.edu/condor 16 How long does this claim exist? › Until it is explicitly preempted  Or on job boundaries if you insist › Who can preempt?  The startd due to resource owner policy  The matchmaker due to organization policy Wants to give the node to a higher priority user  The schedd due to job scheduling policy Stop runaway job, etc. › Responsibility is placed at the appropriate level

17. www.cs.wisc.edu/condor 17 Condor-G Globus 2 Globus 4 Unicore (Nordugrid) Startd Schedd Jobs LSF PBS Schedd - Condor-G - Condor-C -Thanks David @ INFN!

18. www.cs.wisc.edu/condor 18 User/Application/Portal Fabric ( processing, storage, communication ) Grid Condor Pool Middleware (Globus 2, Globus 4, Unicore, …) Condor-G

19. www.cs.wisc.edu/condor 19 Two ways to do the “grid thing” (share resources) › “I’ll loan you a machine, you do whatever you want to do with it”.  This is what Condor does via matchmaking › “Give me your job, and I’ll run it for you”.  Traditional grid broker

20. www.cs.wisc.edu/condor 20 How Flocking Works › Add a line to your condor_config : FLOCK_HOSTS = Site-A, Site-B Schedd Collector Negotiator Matchmaker Collector Negotiator Site-A Matchmaker Collector Negotiator Site-B Matchmaker Submit Machine

21. www.cs.wisc.edu/condor 21 Example of “Give me a job” Broker (Condor-G) Grid Middleware Batch System Front-end Execute Machine

22. www.cs.wisc.edu/condor 22 Any problems? › Sure. Lack of scheduling information Job monitoring details Loss of job semantic guarantees › What can we do? Glide-In Schedd and Startd are separate services that do not require any special privledges  Thus we can submit them as jobs!  Grow a Condor pool out of Grid resources

23. www.cs.wisc.edu/condor 23 Glide-In Startd Example Condor-G (Schedd) Batch System Frontend Middleware Startd Job

24. www.cs.wisc.edu/condor 24 Glide-In Startd › Why?  Restores all the benefits that may have been washed away by the middleware, such as Lack of scheduling information Job monitoring details  End-to-end management solution Preserves job semantic guarantees Preserves policy  Enables lazy planning

25. www.cs.wisc.edu/condor 25 Glide-In Startd (aka ‘sneaky job’) › Why not?  Time wasted on scheduling overhead for black hole-jobs Alternative? Wait-while-idle. Which is more evil? Glide-ins can be removed Glide-in “broadcast” is not the only model  Makes accurate prediction time difficult (impossible?) But can still give worst case › Would not be needed if schedulers could delegate machines instead of just run jobs…

26. www.cs.wisc.edu/condor 26 What’s brewing for after v6.8.0? › More data, data, data  Stork distributed w/ v6.8.0, incl DAGMan support  NeST manage Condor spool files, ckpt servers  Stork used for Condor job data transfers › Virtual Machines (and the future of Standard Universe) › Condor and Shibboleth (with Georgetown Univ) › Least Privilege Security Access (with U of Cambridge) › Dynamic Temporary Accounts (with EGEE, Argonne) › Leverage Database Technology (with UW DB group) › ‘Automatic’ Glideins (NMI Nanohub – Purdue, U of Florida) › Easier Updates › New ClassAds (integration with Optena) › Hierarchical Matchmaking Can I commit this to CVS??

27. www.cs.wisc.edu/condor 27 BIG 10 MM UW MM CS MM Theory Group MM CC R R R R “I need more resources” A Tree of Matchmakers Fault Tolerance Flexibility MM now manage other MMs Erdos MM A Match

28. www.cs.wisc.edu/condor 28 Data Placement * (DaP) must be an integral part of the end-to-end solution Space management and Data transfer *

29. www.cs.wisc.edu/condor 29 Stork › A scheduler for data placement activities in the Grid › What Condor is for computational jobs, Stork is for data placement › Stork comes with a new concept: “Make data placement a first class citizen in the Grid.”

30. www.cs.wisc.edu/condor 30 Stage-in Execute the Job Stage-out Stage-in Execute the jobStage-outRelease input spaceRelease output space Allocate space for input & output data Data Placement Jobs Computational Jobs

31. www.cs.wisc.edu/condor 31 DAGMan DAG with DaP Condor Job Queue DaP A A.submit DaP B B.submit Job C C.submit ….. Parent A child B Parent B child C Parent C child D, E ….. C Stork Job Queue E DAG specification ACB D E F

32. www.cs.wisc.edu/condor 32 Why Stork? › Stork understands the characteristics and semantics of data placement jobs. › Can make smart scheduling decisions, for reliable and efficient data placement.

33. www.cs.wisc.edu/condor 33 Failure Recovery and Efficient Resource Utilization › Fault tolerance  Just submit a bunch of data placement jobs, and then go away.. › Control number of concurrent transfers from/to any storage system  Prevents overloading › Space allocation and De-allocations  Make sure space is available

34. www.cs.wisc.edu/condor 34 Support for Heterogeneity Protocol translation using Stork memory buffer.

35. www.cs.wisc.edu/condor 35 Support for Heterogeneity Protocol translation using Stork Disk Cache.

36. www.cs.wisc.edu/condor 36 Flexible Job Representation and Multilevel Policy Support [ Type = “Transfer”; Src_Url = “srb://ghidorac.sdsc.edu/kosart.condor/x.dat”; Dest_Url = “nest://turkey.cs.wisc.edu/kosart/x.dat”; …… Max_Retry = 10; Restart_in = “2 hours”; ]

37. www.cs.wisc.edu/condor 37 Run-time Adaptation › Dynamic protocol selection [ dap_type = “transfer”; src_url = “drouter://slic04.sdsc.edu/tmp/test.dat”; dest_url = “drouter://quest2.ncsa.uiuc.edu/tmp/test.dat”; alt_protocols = “nest-nest, gsiftp-gsiftp”; ] [ dap_type = “transfer”; src_url = “any://slic04.sdsc.edu/tmp/test.dat”; dest_url = “any://quest2.ncsa.uiuc.edu/tmp/test.dat”; ]

38. www.cs.wisc.edu/condor 38 Run-time Adaptation › Run-time Protocol Auto-tuning [ link = “slic04.sdsc.edu – quest2.ncsa.uiuc.edu”; protocol = “gsiftp”; bs = 1024KB;//block size tcp_bs= 1024KB;//TCP buffer size p= 4; ]

39. www.cs.wisc.edu/condor 39 Planner DAGMan Condor-G Stork RFT GRAM SRM StartD SRB NeST GridFTP Application Parrot

40. www.cs.wisc.edu/condor 40 Some bold assertions for discussion… › Need clean separation between resource allocation and work delegation (scheduling)  Sites just worry about resource allocation › Scheduling policy should be at the VO level and deal with resources first  Don’t ship jobs – ship requests for resources  if you send a job to site, do not assume that it will actually run there.  do not depend on what the site tells you, use your own experience to evaluate how well the site serves you.  Keep in mind that grids are not about running one job, it is about running MANY jobs.  Want CoD? Get your resources first!!

41. www.cs.wisc.edu/condor 41 Some bold assertions for discussion, cont… › Data Handling  Treat data movement as a “first class citizen” › “Lazy Planning” good › Application scheduling “intelligence” should start in a workflow manager and push downwards › Schedulers must be dynamic  Deal gracefully with resources coming and going.

42. www.cs.wisc.edu/condor 42 Thank You! › Questions?