FJPPL meeting Lyon, 17th of February 2010 Sylvain Reynaud

JSAGA2 SAGA specification (overview)

JSAGA3 SAGA specification / JSAGA impl. securitySecuritycontrol protocolDatamonitorExecution

JSAGA4 input data SRM GridFTP LCG-CELCG-CEWS-GRAM firewall Goal: seamless job submission WMS WMS WS-GRAM job desc. gLite plug-ins Globus plug-ins JSAGA staging graph delegate files staging job OPlast EGEE JDLRSL job SAGA middleware heterogeneity (e.g. gLite, Globus, Unicore) how to use the legacy APIs ? JSAGA infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) which security context to use? which transfer path (according to firewalls, protocols, security…) ? which environment variables ? how to use commands available on workers (wget, srmcp, Sget…) ? basedon

JSAGA5 Both implementer & user of SAGA Ready to use software, adapted to targeted scientific field Ready to use software, adapted to targeted scientific field A single interface for using all grid infrastructures A single interface for using all grid infrastructures A single interface for using all middlewares A single interface for using all middlewares As many interfaces as design approaches As many interfaces as design approaches As many interfaces as used technologies As many interfaces as used technologies user applications jobs collection JSAGA SAGA SAGA legacy APIs end user application developer plug-ins developer core engine JSAGA plug-ins JSAGA plug-ins interfaces plug-ins interfaces JSAGA interface implementation

JSAGA6 Plug-ins interfaces in JSAGA Close to application developer needs –object-oriented –high-level –uniform interface for all the supported technologies design objectives –easy to use … but > (T. Kielmann) engine code = 2 x plug-ins code Close to existing middleware APIs –service-oriented –low-level –as many interfaces as design approaches + optional interfaces design objectives –easy to implement –enable efficient usage of middleware APIsplug-insinterfacesJSAGA SAGA SAGA

JSAGA7 Plug-ins interfaces

JSAGA8 planned construction done Job control SAGA user interface: getInput / getOutput Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution gatekeepergLite-WMSwsgramunicore6 set stream for interactive set stream for non- interactive get stream for interactive sshlocal getInput getOutput getError creamnaregi Plug-ins interfaces - job (streams)

JSAGA9 Several ways to monitor jobs –API mode poll job status listen to notifications about job status changes –API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) Plug-in implements several interfaces User see a single interface: SAGA planned construction done Job monitoring SAGA user interface: getState / waitFor Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs query status for individual job listen status for individual job query status for filtered jobs gatekeepergLite-WMSwsgramunicore6 getState waitFor sshlocalcreamnaregi Plug-ins interfaces - job (monitoring)

JSAGA10 Plug-ins interfaces - data planned construction done cache Optimizations: Optional interfaces for plug-ins – –e.g. third-party transfer, recursive find with meta-data Cache management of – –opened connections – –meta-data – –content explicit via plug-in LogicalPhysical rnslfnsrbirodscatalogrbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache

JSAGA11 Plug-ins interfaces - security planned construction done /etc/grid-security/certificates /tmp/x509up_u_$UID ~/.globus/user*.pem X509SSHInMemCredGlobusG. LegacyG. RFC820MyProxyVOMSLogin / pwd proxy path CA certkey p12 proxy pass phrase plug-in supported attributes jsaga-context-init command line

JSAGA12 LogicalPhysical Job monitoring Job control JSAGA plug-ins planned construction done Data Execution Security InMemCredGlobusG. LegacyG. RFC820MyProxyVOMSJKSSSHLogin / pwdX509gatekeepergLite-WMSwsgramunicore6creamnaregisshlocalgatekeepergLite-LBwsgramunicore6creamnaregisshlocal rnslfnsrbirodscatalogrbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache Grid techno. Others

JSAGA13 JSAGA plug-ins SAGA hides most of the differences between the technologies Information on remaining differences is available for each plug-in module… –on JSAGA web site Frequently Asked Questions Configuration example –with command line jsaga-help supported protocols associated security context usage, default attributes usage: jsaga-help -a. | --config | -d | -h | -j | -s | -v [-D. = where: -a,--attribute. Output the value of security context attribute --config Output the effective configuration -d,--data Information about data protocols. = service | context -D. = Set context instance attribute (e.g. -DVOMS[0].UserVO=dteam) -h,--help Display this help and exit -j,--job Information about job services. = service | context -s,--security Information about security context instances. = usage | default | missing -v,--version Output version information and exit

JSAGA14 JSAGA command line interfaces JSAGA provides command line interfaces for… –security jsaga-context-init jsaga-context-info jsaga-context-destroy –execution management jsaga-job-run jsaga-job-status jsaga-job-cancel Also useful as code examples to start with SAGA –data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical

JSAGA15 Software quality Build process fully automated –download and install build tools external libraries –generate source code –execute test-suites unitary tests integration tests –generate project web site documentation reports –generate installer GUI Plug-ins –minimal external dependencies e.g. gLite-UI no needed OS-independent –maven 'archetype' to generate skeleton of new plug-in project –validated by a middleware- independent SAGA test suite # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base= https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/

JSAGA16 Installer GUI

JSAGA17 Licenses LGPL license –for the core engine and for most plug-ins Optional licenses –for plug-ins having external dependencies, which license is not compatible with LGPL –then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the plug-in(s)

JSAGA18 Contributions a web portal for submitting jobs to industrial and research grid infrastructures uses JSAGA to hide grid infrastructure heterogeneity contribution from –many plug-ins Job: gLite-WMS, Unicore 6, Globus WS-GRAM, SSH, local Data: RByteIO, SFTP –automated SAGA execution management test-suite JUX (Java Universal eXplorer) a multi-protocols file browser contribution from P.Calvat –plug-ins for SRB –plug-ins for iRODS JJS (Java Job Submission) a tool for running efficiently short-life jobs on EGEE contribution from P.Calvat –compare overhead with JSAGA / with direct calls to Globus API –scalability tests by JJS users/ logical/physical plug-ins

JSAGA19 Contributions SimExplorer a set of tools, including a workflow engine providing distributed computing facilities, for managing simulation experiments contribution from –scalability tests thousands jobs on many WMS –tests with long-run workflows request + test: auto-reset job service when context renewed –wiki pages jCAE (Java Computer Aided Engineering) Computer Aided Engineering on distributed computing environment contribution from –currently working on SSH plug-in enable controlling and monitoring a SSH job from another JVM –maybe other plug-ins later… GRIA, LSF, torque ?

JSAGA20 Contributions (not a user of JSAGA) contribution from –access to their grid services deployed for D-Grid Unicore 6 Globus –v2 components (Gatekeeper) –v4 components (WS-GRAM) gLite Open source technologies to manage, preserve, and link digital content (upcoming) contribution from P.Y.Jallud –data plug-in to browse content from Fedora Commons repository –will be included in JUX

JSAGA21 Other ideas for contributions New users => new use-cases –new tests bug-fix requests new wiki pages –new ideas feature requests help for testing these features –need support for additional technologies contribute with new plug-ins New plug-ins –for job batch systems (with DRMAA) grids clouds peer-to-peer desktop grids pilot job systems –for data physical/logical file protocols application level protocols –for security SSO (Shibboleth, OpenSSO…)

JSAGA22 Conclusion JSAGA plug-in interfaces designed to –minimize amount of code needed to support new technologies –minimize impact of API uniformity on efficiency and scalability JSAGA plug-in implementations minimize dependencies –on external tools and libraries (e.g. gLite-UI) –on operating system (tested on) JSAGA is opened to external contributions

JSAGA23 Thank you !

JSAGA24 Backup slides

JSAGA25 OpenPlast OpenPlast World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS Globus Globus Transfer path depends on… grid or site – –network filtering policy – –commands available on workers – –services available from workers (close Storage Element, shared FS) – –supported context instances data to stage – –shared by several jobs – –installed on some worker nodes – –file size – –required data protection level execution service – –protocols supported for staging transfer protocol – –access mode (RO, WO, RW) – –third-party transfer – –supported data protection level OPlast GSIFTP EGEE CA SMTPGSIFTPGSIFTPHTTP OPlast SRB job C R1R1R1R1 E1E1E1E1 C' E1E1E1E1 R1R1R1R1 CC' common result std-error

JSAGA26 E1E1E1E1 R1R1R1R1 CC'C'' common EEsrc executable D1D1D1D1 input data result std-error OPlast EGEE CA SMTPGSIFTPGSIFTPHTTP GSIFTP OPlast SRB Transfer path depends on… grid or site –network filtering policy –commands available on workers –services available from workers (close Storage Element, shared FS) –supported context instances data to stage –shared by several jobs –installed on some worker nodes –file size –required data protection level job C"Esrc D1D1D1D1 E C R1R1R1R1 E1E1E1E1 C' TARTAR execution service – –protocols supported for staging transfer protocol – –access mode (RO, WO, RW) – –third-party transfer – –supported data protection level sGet