IGTMD meeting, Lyon Sylvain Reynaud Current status of IGTMD meeting, Lyon Sylvain Reynaud

Use cases User may access several grid infrastructures when he needs… some data available on a given grid a very large number of computing resources resources with specific constraints super-computer confidentiality small overhead (e.g. for consolidation) interactivity … cluster JSAGA

(e.g. gLite, Globus, Unicore) What is JSAGA ? job desc. JDL RSL JSAGA (e.g. gLite, Globus, Unicore) middleware heterogeneity hide gLite plug-ins Globus plug-ins JSAGA

(e.g. gLite, Globus, Unicore) What is JSAGA ? job desc. EGEE OPlast JSAGA (e.g. gLite, Globus, Unicore) middleware heterogeneity hide gLite plug-ins Globus plug-ins JDL RSL staging graph delegate selection & files staging job job WMS input data SRM GridFTP WS-GRAM LCG-CE firewall infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) hide grid JSAGA

Global architecture For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities Class URLFactory introduced in release "1.0rc2". This will enable to implement : support for windows paths support for relative paths protocol-dependant URL encoding … SAGA Plug-ins interfaces JSAGA

Design of plug-in interfaces close to application developer needs object oriented uniform interface to all the supported technologies but << certainly not simple to implement >> (T. Kielmann) code is twice bigger than the code for all the plug-ins close to existing middleware API service oriented as many interfaces as ways to provide the functionalities optional interfaces for technology-specific… optimizations additional features SAGA Plug-ins interfaces Need standard plug-in interfaces to enable to factorize efforts of wrapping middleware API. JSAGA

Plug-in interfaces for execution Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs set stream for interactive set stream for non- interactive get stream for interactive query status for individual job listen status for individual job query status for filtered jobs getInput getOutput getError getState waitFor SAGA user interface: getInput / getOutput SAGA user interface: getState / waitFor Job control Job monitoring gatekeeper gLite-WMS wsgram unicore6 ssh fork cream gridvm PBS proxy gatekeeper gLite-LB wsgram unicore6 ssh fork cream … planned construction done JSAGA

Provided plug-ins Job control Security Data Exec. Exec. Expression Physical files Logical files X509 SSH InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS Login / pwd JKS Data catalog rns lfn srb / irods http https sftp rbyteio file zip gsiftp tar ftp mail cache srm Exec. (control) Exec. (monitor) Job control gatekeeper gLite-WMS wsgram unicore6 ssh local cream gridvm PBS remote gatekeeper gLite-LB wsgram unicore6 ssh local cream … Expression Language basic default JEP BeanShell JSDL+ext. SAGA JDL RSL-2 RSL-4 planned construction done JSAGA

Plug-ins out of SAGA scope piece of a sample JSDL job description in JDL description in JSDL description in RSL <POSIXApplication> <Executable>myexec</Executable> <Argument>-from @{date('dd/MM/yyyy', now+24*INDEX)}</Argument> </POSIXApplication> … Job Collection Job JSDL retry rules termination parametric language specific data staging extensions <Executable>myexec</Executable> <Argument>-from 30/05/2008</Argument> <POSIXApplication> </POSIXApplication> … <Executable>myexec</Executable> <Argument>-from 31/05/2008</Argument> <POSIXApplication> </POSIXApplication> … <Executable>myexec</Executable> <Argument>-from 01/06/2008</Argument> <POSIXApplication> </POSIXApplication> … Expression Language basic default JEP BeanShell JSDL+ext. SAGA JDL RSL-2 RSL-4 planned construction done JSAGA

(e.g. gLite, Globus, Unicore) Demo (part 1) hide middleware heterogeneity (e.g. gLite, Globus, Unicore) Using SAGA API security context data management execution management Gatekeeper Unicore 6 SSH fork export PATH="$PATH:$PWD/bin" jsaga-context-init.bat Globus jsaga-context-info.bat jsaga-ls.bat file://c:/ jsaga-ls.bat http://grid.in2p3.fr/archives/jsaga/ jsaga-cat.bat http://grid.in2p3.fr/archives/jsaga/maven-metadata.xml cd doc/examples/test_demo1/ jsaga-job-run.bat -Executable f:/cygwin/bin/date.exe -Arguments --utc –r <resource> JSAGA

Global architecture For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities Job Collection SAGA Plug-ins interfaces JSAGA

Describe grid infrastructures requirements VOMS Hide middleware heterogeneity Uniform interface (SAGA) Hide infrastructure heterogeneity Selection of the right security context Selection of the right client API configuration Transport of the job input/output data to/from worker nodes Describe the capabilities of the infrastructures you want to use CC-IN2P3 WMS gsiftp:// EGEE VOMS wsgram OpenPlast Grid Globus localhost World JSAGA

Describe grid infrastructures examples VOMS gatekeeper Hide middleware heterogeneity Job management technologies e.g. CREAM, WMS, SSH, GK Hide infrastructure heterogeneity Computing Elements (e.g. GK) Different grid or site policies e.g. network filtering, shared FS Different environment variables e.g. $VO_?_SW_DIR, /usr/local Different configuration attributes e.g. monitor service URL, shell path on cygwin, default SE URL Commands available on worker e.g. globus-url-copy, srmcp, Scp, wget, tar srb:// srm:// CC-IN2P3 WMS lfn:// gsiftp:// EGEE VOMS gatekeeper wsgram OpenPlast Grid http:// Globus tar:// gatekeeper localhost World JSAGA

Transfer path depends on… World Grid EGEE OpenPlast localhost CC-IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper wsgram WMS VOMS Globus Using a single grid all files can be transferred to the worker through a single node Using several grids need to dynamically build a transfer graph, according to… url:// job desc. JSAGA plug-ins JSAGA

Transfer path depends on… grid or site network filtering policy commands available on workers services available from workers (close Storage Element, shared FS) supported context instances execution service protocols supported for staging transfer protocol access mode (RO, WO, RW) third-party transfer supported data protection level data to stage shared by several jobs installed on some worker nodes file size required data protection level World Grid EGEE OpenPlast localhost CC-IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper wsgram WMS VOMS Globus url:// job desc. JSAGA plug-ins JSAGA

Transfer path depends on… C C' common result std-error Transfer path depends on… grid or site network filtering policy commands available on workers services available from workers (close Storage Element, shared FS) supported context instances execution service protocols supported for staging transfer protocol access mode (RO, WO, RW) third-party transfer supported data protection level data to stage shared by several jobs installed on some worker nodes file size required data protection level OPlast EGEE CA SMTP GSIFTP HTTP SRB C R1 E1 C' OpenPlast OPlast GSIFTP job World Grid EGEE OpenPlast localhost CC-IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper wsgram WMS VOMS Globus OPlast OpenPlast JSAGA

Transfer path depends on… C C' C'' common Transfer path depends on… E src executable D1 input data R1 result grid or site network filtering policy commands available on workers services available from workers (close Storage Element, shared FS) supported context instances execution service protocols supported for staging transfer protocol access mode (RO, WO, RW) third-party transfer supported data protection level data to stage shared by several jobs installed on some worker nodes file size required data protection level E1 std-error OPlast EGEE CA SMTP GSIFTP HTTP SRB C R1 E1 C' D1 E src C" OPlast job OPlast E TAR TAR iGet JSAGA

(e.g. gLite, Globus, Unicore) Demo (part 2) hide middleware heterogeneity (e.g. gLite, Globus, Unicore) hide grid infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) Using SAGA API security context data management execution management Gatekeeper Unicore 6 SSH fork Using Job Collection API execution mgt. EGEE OpenPlast my laptop World Grid EGEE OpenPlast localhost CC-IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper wsgram WMS VOMS Globus vi etc/jsaga-engine.properties more doc/examples/jsaga-config.xml jsaga-help.bat --config | more cd doc/examples/test_demo2/ more job.xml jsaga-jobcollection-run.bat job.xml -r resources.xml –dump-wrapper MYJOB_2 | more jsaga-jobcollection-run.bat job.xml -r resources.xml --graph --name all # ... --name openplast_sbx # ... --name openplast # ... --name egee JSAGA

Software quality Fully automated build process download build tools, modules generate code, build, run tests Generated project web site : http://grid.in2p3.fr/jsaga/ documentation user guides (from annotations) specification (from code) design (from code) reports static code analysis test results and coverage tasks list Provide wizard to generate skeleton of new plug-in projects Provide reusable test suite for validating plug-ins # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy lfn.base=lfn:///grid/dteam/ lfn.context=EGEE_proxy http.base=http://grid.in2p3.fr/html/Public/ file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

Software quality Minimize dependencies Provide an installer resolve conflicts between versions of libraries with maven regenerate stubs from WSDL with Axis 1.4 remove external dependencies no gLite-UI for EGEE, no OS-specific dependency Provide an installer install only what you need… JSAGA

Software metrics Specification Implementation SAGA, version 1.0 (January 15, 2008) : 323 pages Official Java Binding, version "snapshot-1" Implementation 46 third-party libraries (only 5 are for the core engine) 644 java classes (+ 508 generated java classes) some code in other languages C XSL XML Schema LL grammar Bash shell MS-DOS batch JSAGA

Main assets of JSAGA does not sacrifice scalability and efficiency for abstraction thanks to design (in particular the definition of plug-in interfaces) thanks to caching mechanisms does not only hide heterogeneity of middleware, but also hides heterogeneity of grid infrastructures thanks to use existing grid infrastructures as they are World Grid EGEE OpenPlast localhost CC-IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper wsgram WMS VOMS Globus plug-ins JSAGA url:// job desc. JSAGA

Global architecture For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities Applications Job Collection SAGA Plug-ins interfaces JSAGA

/ Related projects JSAGA is used by… Applications Elis@ a web portal for submitting jobs to industrial and research grid infrastructures JJS (Java Job Submission) a tool for submitting job collections to EGEE optimized for short-life jobs (resource selection based on QoS observed while submitting jobs) JUX (Java Universal eXplorer) a multi-protocol file browser is currently testing it / JSAGA

Perspectives Improvement of the core engine (IGTMD project) move to SAGA Java Binding release candidate 2 (and next...) bug-fixes, new features, etc. Development of plug-ins (contribution to other projects) RA-Grid plug-ins for OAR, CiGri NAREGI plug-ins for Super Scheduler, (GridVM ?) for BQS users who want to progressively use grid plug-ins for BQS JJS plug-ins for CREAM, LCG-CE (extends GK plug-in), finish plug-ins for SRM, LFC plug-ins for JDL, tar etc. JSAGA