EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMPROXY usage Álvaro Fernández IFIC (CSIC)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Contents Introduction to WMPROXY Architecture of WMPROXY inside WMS Features and JDL attributes APIs: Issues and future work

Enabling Grids for E-sciencE EGEE-II INFSO-RI Introduction gLite idea was to move towards a Service Oriented Architecture (SOA) Imply services communicate through the exchange of messages –Moving to WS-* interfaces –Still missing a real standard. Many WS-* specifications –Activity inside OGF-GIN WMS started moving the access services: –(“old”) Network server receives user requests from UI, using an own protocol and interface. Communication from UI based on provided cli and apis –WMProxy service is web based and provides an standard interface to manage jobs.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim WMPROXY features Workload Manager Proxy (WMProxy) –Developed by Datamat and included in the WMS –Web Service Interface  Simplicity/Extensibility  Accessibility: greater community of users  Integration: Eases interoperability among different middlewares –Stepping towards Interoperability  SOA conformance: WSDL service description  WS-I compliance: support for multi-language, multi-platform clients –Features  Provide Access to gLite Workload Management System  Serve a large number of requests  Improve performances  Improve usability  Provide new functionalities

Enabling Grids for E-sciencE EGEE-II INFSO-RI Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim WMProxy: Architecture (1/4) Service Container: –Apache + GridSite + FastCGI SOAP tooling –Stub generation is performed through gSOAP (C++) Security: –SSL Authentication –DN / FQAN-based Authorization Operations: –Job Submission and Control services implementation (C++) –Gridsite Delegation service implementation (C++)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim WMProxy: Architecture (2/4) gSOAP layer: –intermediate layer between gSOAP and the WMProxy core Directory Manager: –Job reserved area Creation/Management on the server file system LB Access: –interaction with LB components –job registration –logging for job events –job information queries Request Delivery: –deliver request to Workload Management

Enabling Grids for E-sciencE EGEE-II INFSO-RI Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim WMProxy: Architecture (3/4) WMProxy integration with the WMS LB Proxy Workload Manager Client Local File System LB Data Base Server Host Logging & Bookkeeping MOD SSL MOD FastCGI WMProxy Server Apache Request Queue SOAP/ HTTPS Gridsite Delegation Job Sub/Contr.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Taken from: M.Pappalardo - EuChina Grid Interoperability Meeting, Aprim WMProxy: Architecture (4/4) WMProxy modules Client WMProxy Operations LB AccessDelegation Request Delivery Directory Manager AuthorizationgSOAP Layer gSOAP Independant WMProxy Server

Enabling Grids for E-sciencE EGEE-II INFSO-RI Usage: Command Line Interface Interface is described in the WSDL Command line interface based on the that one to provide access: glite-wms-job-submit submits a job to a WMProxy Service. It requires a JDL file as input and returns a WMS job identifier. glite-wms-delegate-proxy allows the user to delegate her proxy credential to the WMProxy service. –This delegated credential can then be used for job submissions. glite-wms-job-list-match lists the identifiers of jobs submitted to a WMProxy Service by the user issuing the command. glite-wms-job-cancel cancels one or more jobs previously submitted to WMProxy Service. glite-wms-job-output Retrieve output files of a job, when finished. After this operation the job itself is purged and no more operation can be done on it glite-wms-job-perusal Manage the perusal functionality for a certain job such us: –Enabling perusal for one (or more) file(s) - Allowing perusal file retrieval - Disabling perusal functionality glite-wms-job-info (new in 3.1): Allow retrieving useful information about the user delegated proxy, the delegated identification details of the user, the jdl of a previously submitted job, etc

Enabling Grids for E-sciencE EGEE-II INFSO-RI Usage Features Credential delegation (glite-wms-delegate-proxy) is used to make the security framework lighter: –Time consuming proxy delegation is done once per session –Job submission uses the delegation id to send jobs –Still can be delegated with each job-submission (as before) – not recommended Types of Requests: –JOB, DAG, COLLECTION Types of supported JOB: –NORMAL, INTERACTIVE, MPICH, PARTITIONABLE, CHECHKPOINTABLE, PARAMETRIC Bulk job submission: enables submission of multiple jobs at the same time –Reduces time of submission –Single call to WMProxy server –Single Authentication and Authorization process –Sharing of files between jobs Job’s Sandboxes: share common files Asynchronous Job Start

Enabling Grids for E-sciencE EGEE-II INFSO-RI Examples Taken from: TEGEE-JRA1-TEC JDL-Attributes-v0-8T 11 Features: JDL Attributes Inclusion of pre- and post- execution scripts: –PROLOGUE, EPILOGUE –PROLOGUEARGUMENTS, EPILOGUEARGUMENTS Compression and expansion of input sandbox files: –ZippedISB = “BossArchive_1_2_1.tgz”; –AllowZippedISB = true; (overrides ZippedIsb with the input sandbox files) Define how much time should the job be maintained in queue for execution: –ExpiryTime = ; Job Perusal: obtain partial results of the output files before job finishes: –PerusalFileEnable = true; –PerusalTimeInterval = 10; –PerusalFilesDestURI = “gsiftp://ghemon.cnaf.infn.it/home/glite/peek”; (not mandatory, could use the glite-wms-job-perusal command to retrieve files)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Requirements JDL Attributes Sandbox files can be stored at a https/gridftp server: –InputSandboxBaseURI = "gsiftp://matrix.datamat.it:5432/tmp"; or multiple servers:  OutputSandbox = { "myjobOutput","run1/event1","myjobError"};  OutputSandboxDestURI = { "gsiftp://matrix.datamat.it:5432/tmp/myjobOutput ", "gsiftp://grid003.ct.infn.it:6789/home/cms/event1", "myjobError“}; DataRequirements contains 3 attributes: –InputData: List of InputData = { “lfn:/EO.test.file”, “lds:cms.test.file”, “guid:135b7b23-4a6a-11d7-87e7-9d101f8c8b70”, “query:select_my_DC1_files” }; –DataCatalogType (“rls”, “si” storage-index, “dli” data location interface) –DataCatalog (addess of catalog - DataCatalog = ); Matchmaking to find a CE close to the SE: –OutputSE = “grid001.cnaf.infn.it";

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Requirements Example DataRequirements = { [ DataCatalogType = "DLI"; DataCatalog = " InputData = {"lfn:/my/test/data1", "guid:44rr44rr77hh77kkaa3", "lds:my.test.dataset", "query:my_query"}; ], [ DataCatalogType = "SI"; DataCatalog = " InputData = {"lfn:/eo/test.file", "guid:ddffrg5451"}; ], [ DataCatalogType = "RLS"; DataCatalog = " InputData = {"lfn:/atlas/test.file", "guid:ggrgrg5656"}; ], [ DataCatalogType = "RLS"; InputData = {"lfn:/myvo/test.file", "guid:adbdefgilm1234"}; ],.... };

Enabling Grids for E-sciencE EGEE-II INFSO-RI Other Features Shallow resubmision: when submission fails due to a mw failure, and not application fail. –ShallowRetryCount = 3; Deep resubmision: RetryCount = 0; –shallow retry count is reset to zero each time a deep resubmission of the job is done. Collections: set of independent jobs –Attributes like a DAG, but no Dependencies among nodes set. nodes = { [n1], [n2], …};

Enabling Grids for E-sciencE EGEE-II INFSO-RI API Usage Command line interface API: provided wsdl, and generated bindings: –Org.glite.wms.interface –Org.glite.wms.wmproxy-api-cpp –Org.glite.wms.wmproxy-api-java –Org.glite.wms.wmproxy-api-python –… API functionality –delegating the credential ; –registering and submitting jobs ; –cancelling the job during its life-cycle ; –retrieving information on the location where the job input sandbox files can be stored ; –retrieving the output sandbox files list ; –retrieving a list of possible matching Computer Elements ; –getting JDL templates ; –getting information on the user disk quota on the server Credential delegation is done by external packages. Only Java include bindings to perform it from the api. (C++ and python users should do it explicitely) In case of file transfer needed, job should be previously registered: –jobIds = client.jobRegister(jdlString, delegationId)

Enabling Grids for E-sciencE EGEE-II INFSO-RI API Usage (cont) File transfers from UI WMS should be done also explicitely in the code: –JAVA: In order to allow grid user to invoke, within his Java code, the UrlCopy class to copy files from the WMProxy Server to the UI and vice versa the following Java package is requested cog-jglobus-1.4- bin.tar.gz The above package with all the needed documentation can be downloaded from this web page New in glite3.1: getDelegatedProxyInfogetDelegatedProxyInfo(java.lang.String delegationId) Returns the Delegated Proxy information identified by the delegationId string getJobProxyInfogetJobProxyInfo(java.lang.String jobId) Returns the information related to the proxy used to submit a job that identified by its JobId. getJDLgetJDL(java.lang.String jobId, org.glite.wms.wmproxy.JdlType type) Returns the JDL string which identifier is the input JobId

Enabling Grids for E-sciencE EGEE-II INFSO-RI Issues Performance was degrading when submitting collections, bulk submission. Solved changing hanling from DAGman to native methods. API: Changing implementation of soap python implementation from SOAPpy to ZSI (see Patch#843)Patch#843 Python bindings: Developers said Delegation functionality that is needed is only provided by other package (org.glite.data.delegation-api-c) implemented in C.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Thanks! For more info: age