Sylvain Reynaud, Pascal Calvat CC-IN2P3 Grid interoperability using

JSAGA2 Plan demo of overview of demo of summary and perspectives JUX JSAGA is an API for uniform access to grids. JJS and JUX are tools using JSAGA.

JSAGA3 23/04/ JJS – Overview JJS was developed by Pascal Calvat (CC-IN2P3) in 2003, to submit jobs to the DATAGRID infrastructure –has evolved to submit jobs to the EGEE infrastructure JJS is designed to ease job submission from web servers hosted in laboratories –it is an alternative to User Interface + Resource Broker (or to gLite- UI + gLite-WMS) JJS is optimized for submitting short-life jobs –based on observed QoS of sites: JJS give a score to selected sites and use it for subsequent match-makings –but it can also be used with long-life jobs

JSAGA4 23/04/ JJS – Demo Execution time on local host Execution time on EGEE grid infrastructure Ratio 8000 s600 s with 100 jobs13 Overall performance for short-life jobs (install povray on-the-fly, then generate part of the image) 1 job

JSAGA5 23/04/ JJS – Overview JJS was initially developed on top of cog-jglobus API cog-jglobus is being replaced with JSAGA for… –security(done) –data management(done) –execution management(in a near future) –job collection management(in a near future) Using JSAGA enables JJS to become independent of gLite middleware evolutions –from Globus proxy to VOMS proxy(done) –from GSIFTP to SRM(work in progress…) –from LCG-CE to gLite-CREAM(in a near future)

JSAGA6 cluster JSAGA – targeted use cases Motivations for using several grid infrastructures: increasing the number of computing resources available to user need for resources with specific constraints super-computer confidentiality small overhead (e.g. consolidation) interactivity availability, on a given grid, of: the data the software

JSAGA7 Ready-to-use software, adapted to targeted scientific field Ready-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructures Hide heterogeneity between grid infrastructures Hide heterogeneity between middlewares Hide heterogeneity between middlewares As many interfaces as ways to implement each functionality As many interfaces as ways to implement each functionality As many interfaces as used technologies As many interfaces as used technologies SAGA SAGA

JSAGA8 SAGA: code example // use factories to create SAGA objects Session session = SessionFactory.createSession(); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); NSDirectory dir = NSFactory.createNSDirectory(session, url); // use SAGA objects List result = dir.list(); for (URL r : result) System.out.println(r); SAGA SAGA

JSAGA9 Ready-to-use software, adapted to targeted scientific field Ready-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructures Hide heterogeneity between grid infrastructures Hide heterogeneity between middlewares Hide heterogeneity between middlewares As many interfaces as ways to implement each functionality As many interfaces as ways to implement each functionality As many interfaces as used technologies As many interfaces as used technologies core engine + plug-ins JSAGA end user application developer plug-ins developer SAGA SAGA

JSAGA10 Plug-ins interfaces close to application developer needs –object-oriented –high-level –uniform interface to all the supported technologies design objectives –easy to use … but > (T. Kielmann) engine code = 2 x plug-ins code close to existing middleware APIs –service-oriented –low-level –as many interfaces as ways to implement each functionality –optional interfaces design objectives –easy to implement –enable efficient usage of middleware APIs plug-insinterfaces JSAGA core engine + plug-ins JSAGA SAGA SAGA

JSAGA11 Plug-ins: execution management planned construction done Job control SAGA user interface: getInput / getOutput Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution gatekeepergLite-WMSwsgramunicore6 set stream for interactive set stream for non- interactive get stream for interactive sshfork getInput getOutput getError creamPBSremotenaregi Job monitoring SAGA user interface: getState / waitFor Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs query status for individual job listen status for individual job query status for filtered jobs gatekeepergLite-LBwsgramunicore6 getState waitFor sshforkcream… core engine + plug-ins JSAGA

JSAGA12 planned construction done Plug-ins provided Data Expression catalogrnslfnsrb / irodshttphttpssftprbyteiofilezipgsiftptarftpmailcachesrm basicdefaultJEPBeanShell Language JSDL+ext.SAGAJDLRSL-2RSL-4 Exec.(control) Job control gatekeepergLite-WMSwsgramunicore6sshforkcreamPBSremotenaregi Exec.(monitor) gatekeepergLite-LBwsgramunicore6sshforkcream… Security InMemCredGlobusG. LegacyG. RFC820MyProxyVOMS Logical files core engine + plug-ins JSAGA X509SSHLogin / pwdJKS Physical files

JSAGA13 This is still not enough… job desc. gLite plug-ins Globus plug-ins JSAGA hidemiddlewareheterogeneity (e.g. gLite, Globus, Unicore) JDLRSL core engine + plug-ins JSAGA

JSAGA14 WMS WMS input data SRM GridFTP WS-GRAM LCG-CELCG-CEWS-GRAM firewall This is still not enough… job desc. gLite plug-ins Globus plug-ins JSAGA job staging graph delegate selection & files staging job OPlast EGEE hideinfrastructuresheterogeneity (e.g. EGEE, OSG, DEISA) hidemiddlewareheterogeneity (e.g. gLite, Globus, Unicore) JDLRSL

JSAGA15 Ready-to-use software, adapted to targeted scientific field Ready-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructures Hide heterogeneity between grid infrastructures Hide heterogeneity between middlewares Hide heterogeneity between middlewares As many interfaces as ways to implement each functionality As many interfaces as ways to implement each functionality As many interfaces as used technologies As many interfaces as used technologies core engine + plug-ins JSAGA jobscollection JSAGA end user application developer plug-ins developer SAGA SAGA

JSAGA16 Description of infrastructures Middleware heterogeneity –e.g. CREAM, WMS, SSH, GK Infrastructures heterogeneity –Grid/site policy e.g. network filtering, shared FS –Environment variables e.g. $VO_?_SW_DIR, /usr/local –Configuration attributes (client) e.g. monitor service URL, shell path on cygwin, default SE URL –Command line interfaces (worker) e.g. globus-url-copy, srmcp, Scp, wget, tar World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus example: execution management jobscollection JSAGA

JSAGA17 url:// Transfer path depends on… When using a single grid infrastructure –all files can be transported to/from the worker nodes through a single storage node When using several grid infrastructures –need to dynamically build a more complex transfer graph, according to… jobdesc.plug-insJSAGA jobscollection JSAGA World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus

JSAGA18 Transfer path depends on… grid or site – –network filtering policy – –commands available on workers – –services available from workers (close Storage Element, shared FS) – –supported context instances data to stage – –shared by several jobs – –installed on some worker nodes – –file size – –required data protection level execution service – –protocols supported for staging transfer protocol – –access mode (RO, WO, RW) – –third-party transfer – –supported data protection level jobdesc. World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus url://plug-insJSAGA jobscollection JSAGA

JSAGA19 jobscollection JSAGA OpenPlast OpenPlast World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus Transfer path depends on… grid or site – –network filtering policy – –commands available on workers – –services available from workers (close Storage Element, shared FS) – –supported context instances data to stage – –shared by several jobs – –installed on some worker nodes – –file size – –required data protection level execution service – –protocols supported for staging transfer protocol – –access mode (RO, WO, RW) – –third-party transfer – –supported data protection level OPlast GSIFTP EGEE CA SMTPGSIFTPGSIFTPHTTP OPlast SRB job C R1R1R1R1 E1E1E1E1 C' E1E1E1E1 R1R1R1R1 CC' common result std-error

JSAGA20 jobscollection JSAGA E1E1E1E1 R1R1R1R1 CC'C'' common EEsrc executable D1D1D1D1 input data result std-error OPlast EGEE CA SMTPGSIFTPGSIFTPHTTP GSIFTP OPlast SRB Transfer path depends on… grid or site –network filtering policy –commands available on workers –services available from workers (close Storage Element, shared FS) –supported context instances data to stage –shared by several jobs –installed on some worker nodes –file size –required data protection level job C"Esrc D1D1D1D1 E C R1R1R1R1 E1E1E1E1 C' TARTAR execution service – –protocols supported for staging transfer protocol – –access mode (RO, WO, RW) – –third-party transfer – –supported data protection level iGet

JSAGA21 jobscollection JSAGA Example of generated graph Data flow E1E1E1E1 R1R1R1R1 CC'C'' common EEsrc executable D1D1D1D1 input data result std-error OPlast example with several protocols used, but only 3 jobs submitted on 1 grid…

JSAGA22 Ready-to-use software, adapted to targeted scientific field Ready-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructures Hide heterogeneity between grid infrastructures Hide heterogeneity between middlewares Hide heterogeneity between middlewares As many interfaces as ways to implement each functionality As many interfaces as ways to implement each functionality As many interfaces as used technologies As many interfaces as used technologies Applications end user application developer plug-ins developer core engine + plug-ins JSAGA jobscollection JSAGA SAGA SAGA

JSAGA23 Command line interfaces JSAGA provides command line interfaces for… –security jsaga-context-init jsaga-context-info jsaga-context-destroy –execution management jsaga-job-run jsaga-job-status jsaga-job-cancel –data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical Applications Applications

JSAGA24 Related projects JSAGA is used by… a web portal for submitting jobs to industrial and research grid infrastructures –JJS (Java Job Submission) a tool for submitting jobs to EGEE optimized for short-life jobs (resource selection based on QoS observed while submitting jobs) –JUX (Java Universal eXplorer) a multi-protocols file browser Applications / Applications

JSAGA25 JUX – Overview JUX is a file explorer designed to be independent of –Operating System tested on Windows, Scientific Linux, Ubuntu, Mac –Data management protocol tested with gsiftp, srb, irods, http, https, sftp, zip, (srm) –Security mechanism tested with GSI, VOMS, Login/Password, X509, SSH –File content viewer provided viewers are for text file, image viewer, audio player can use local applications (only for protocol "file://" on OS "Windows") full java code JSAGA png, gif, jpg, bmp, tiff, dicom mp3, wav

JSAGA26 JUX – Overview Data management and security –JUX does not only use the SAGA API –it also uses the JSAGA introspection API to discover… list of available protocols list of configured security contexts list supported security context types, for each protocol –this allows JUX to be completely independent of technologies used just copy your own JSAGA plug-in in JUX "lib/" directory to add the support for a new technology !

JSAGA27 Demo of JUX … and then conclusion about

JSAGA28 Software quality Build process fully automated, including… –build tools installation –code generation –testing unitary tests integration tests –project web site generation –installer GUI generation (see next slide…) Plug-ins –external dependencies reduced e.g. gLite-UI not needed most plug-ins supports –a maven 'archetype' generates skeleton of new plug-in project –plug-ins automatically validated with a reusable SAGA test suite # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base= https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/

JSAGA29 Installer GUI

JSAGA30 License(s) LGPL license –for the core engine and most plug-ins Optional licenses –for plug-ins having external dependencies, which license is not compatible with LGPL –then, end-user must… either accept the terms of the license agreement or uncheck these plug-ins (see previous slide)

JSAGA31 Summary Main assets of JSAGA Implement standard specifications from –SAGA –JSDL Provide high-level abstraction layer with no sacrifice on efficiency or scalability –thanks to design (definition of plug-ins interface) –thanks to cache mechanisms Use grid infrastructures as they are (i.e. no pre-requisite) –thanks to Hide heterogeneity middlewares –of middlewares grid infrastructures –of grid infrastructures World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus

JSAGA32 Perspectives Support new technologies –develop plug-ins gLite-CREAM French research grid middleware ? … –integrate plug-ins developed by partners Implement new specifications –SAGA Extension: Service Discovery API discussions on candidate spec. has just finished, the final spec. should be available soon JSAGA –has no equivalent for this –plug-in based implementation –JSDL Extension: Parameter Sweep Job proposed for public comments JSAGA does this in a non- standard way

JSAGA33 Backup slides

JSAGA34 JUX Plan overview summary and perspectives overview summary and perspectives overview summary and perspectives

JSAGA35 23/04/ JJS – Performance For short-life jobs, grid overhead is not negligible  need to optimize each step of job submission: → job submission:multi-threaded → data staging:input/output files are grouped in tarballs → monitoring:get all job status with a single request → job life-time:waiting and running jobs have a timeout limit …and last but not least: select the execution sites, which are the most efficient for short-life jobs (based on observed QoS)

JSAGA36 23/04/ JJS – Performance (submission) Average time before entering state WAITING 12 seconds 95% of jobs enter state WAITING before… 15 seconds Time elapsed before entering state WAITING (i.e. time for transferring the input sandboxes + submitting the jobs)

JSAGA37 23/04/ JJS – Performance (monitoring) StepFile name extensionJob status Input sandbox uploaded.tarUPLOADED Job submitted to CE(.tar)WAITING Job started.runRUNNING Job completed.res.tarDONE Average time for getting status of all jobs 3 seconds Use naming convention on GSIFTP server instead of Globus monitoring (detecting job failure is not needed because all the jobs timeout shortly…)

JSAGA38 23/04/ JJS – Summary Optimized for short-life jobs –QoS-based selection of execution sites –pragmatic usage of deployed grid technologies Easy to install, configure and use Robust –designed to be not sensible to grid middleware failures –because developed when grid was not mature (DATAGRID)

JSAGA39 JJS - Perspectives Finish integration of JSAGA –for job submission (SAGA) –for job collection management (JSDL Parameter Sweep Job Extension) job description: independent of language data staging: independent of protocols and infrastructure constraints JJS is also waiting… –for SRM data management JSAGA plug-in –for Service Discovery API (SAGA Extension) support in JSAGA in order to enable efficient usage of SRM with short-life jobs (by discovering GSIFTP servers through the SRM web service)

JSAGA40 Plan overview summary and perspectives overview summary and perspectives overview summary and perspectives JUX

JSAGA41 JUX – Screenshots 23/04/ The connection manager enables user to create connection profiles with URL and security context. Only the security contexts compatible with selected protocols appear in the popup list.

JSAGA42 JUX – Screenshots 23/04/ Connection is kept open until the nodes are collapsed (left side). Copy several files with a single drag-and-drop.

JSAGA43 JUX – Related work Similar tools exist –HERMES (Australia) –VBrowser (Holland) Using JSAGA for JUX enables –to factorize development efforts with JJS (for data staging) –to manage logical files through a common interface (SAGA) –protocol-specific optimizations e.g. third-party transfer, filtered file list –to automatically recover some errors e.g. create parent directory if missing, retry if error is IncorrectState based on Apache Commons VFS

JSAGA44 JUX – Summary JUX can work with potentially any –protocol –security mechanism –file content JUX is easy to use –targeted users are scientists JUX is lightweight –currently 11 MB with all plug-ins you can develop the plug-ins missing for your use-case

JSAGA45 JUX – Perspectives (meta-data) NameValue DICOM Study Date18/11/2008 DICOM Patient's NameJohn Smith DICOM Patient's SexM DICOM Patient's Age28 size

JSAGA46 JUX – Perspectives (meta-data) SEARCH Study Date Patient's Name John S*  Recursive *.txt entry name Search M Patient's Sex Patient's Age size and