1. 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk

2. 2 Background

3. 3 Motivations  Adding Web Service interfaces to Condor daemons  Facilitate the development of third-party applications capable of interacting with Condor (remotely). –E.g. build higher-level application specific scheduler that submits jobs to multiple Condor pools based on application semantics –These can be built using a wide range of languages/SOAP packages –BirdBath has been tested on: Java (Apache Axis, XSUL) Python (ZSI) C# (.Net) C/C++ (gSOAP)  Condor accessible from platforms where its command-line tools are not supported/installed

4. 4 The (Hypothetical) Big Picture Schedd flocking Schedd SOAP Globus/ gridSAM Condor-G Client Site ASite BSite C Site D SOAP

5. 5 Condor SOAP Interfaces Central manager Execution machine Submission machine(s) Collector Schedd Execution machine Execution machine Execution machine  Begin/commit/abort transaction  Submit Job  Obtain Job ClassAds  Get/Send file  Removed/hold/release job  Obtain resource ClassAds  Obtain all daemon ClassAds Shadow Negotiator  Now available in Condor 6.7.5

6. 6 Clients for BirdBath  WSDLs available that describe each daemon interface  SOAP libraries (stub code) can be generated automatically from WSDLs Custom Client holdJob() SOAP library Submission machine(s) Schedd

7. 7 Example: Query Collector  Obtain information (ClassAds) about all resources that have over 512MB RAM collectorLocator = new CondorCollectorLocator(); collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”); classAds = collector.queryStartdAds(“Memory>512”);

8. 8 Example: Job Submission 1.Create New Transaction 2.Create New JobId 3.Send Files ({input files, binaries}) 4.Create Job Description 5.Submit Job 6.Commit Transaction  Transaction based process  Uses two phase commit to promote reliability and robustness  Files transferred in chunks – Base64 encoded SOAP attachments  DAGMan jobs supported

9. 9 OMII Collaboration: gridSAM plugin  Job Submission & Monitoring Web Service that submits to range of resource managers (e.g. Condor, SGE, Globus)  Uses JSDL (Job Submission Description Language) for standardised job description GridSAM Condor SGE Fork … … Client SOAP JSDL /bin/echo hello world GridSAM Condor SGE Fork … … Client SOAP JSDL

10. 10 gridSAM plugin  Existing Condor plugin uses shell-based submission via command- line tools and monitoring of log files GridSAM Condor SGE Fork … … Client Condor-WS  Our plugin accesses Condor via the SOAP interfaces Submission machine(s) Schedd Middle-tier JSDL SOAP ClassAd  Building JSDL to ClassAd conversion libraries

11. 11 Future Work  Bundle plug-in into gridSAM –Also provide separate JSDL to ClassAd conversion library  Extensions to other daemons (what functionality to expose?) –For example, allow checkpoint data to be retrieved through the shadow in order to migrate jobs across pools  Roll out daemons on e-minerals mini-grid