EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks 4th EGEE User Forum Catania, 3 march 2009 Yonny CARDENAS (on behalf of Sylvain REYNAUD) Uniform access to grid infrastructures with

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 2 cluster Motivations for using several grid infrastructures: increasing the number of computing resources available to user need for resources with specific constraints super-computer confidentiality small overhead (e.g. consolidation) interactivity availability, on a given grid, of: the data the software JSAGA – targeted use cases

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 3 Ready-to-use software, adapted to targeted scientific fieldReady-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructuresHide heterogeneity between grid infrastructures Hide heterogeneity between middlewaresHide heterogeneity between middlewares As many interfaces as ways to implement each functionalityAs many interfaces as ways to implement each functionality As many interfaces as used technologiesAs many interfaces as used technologies SAGA SAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 4 Simple API for Grid Applications SAGA enables hiding heterogeneity between middlewares –a uniform interface, in particular for  security  data management  execution management –OGF standard Example with data mgt (ls): SAGA SAGA // use factories to create SAGA objects Session session = SessionFactory.createSession(); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); NSDirectory dir = NSFactory.createNSDirectory(session, url); // use SAGA objects List result = dir.list(); for (URL r : result) System.out.println(r);

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 5 Ready-to-use software, adapted to targeted scientific fieldReady-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructuresHide heterogeneity between grid infrastructures Hide heterogeneity between middlewaresHide heterogeneity between middlewares As many interfaces as ways to implement each functionalityAs many interfaces as ways to implement each functionality As many interfaces as used technologiesAs many interfaces as used technologies core engine + plug-ins JSAGA end user application developer plug-ins developer SAGA SAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 6 Plug-ins interfaces design objectives –easy to use … but > (T. Kielmann)  amount of code for engine = 2 x amount for all plug-ins close to application developer needs –object-oriented –high-level –uniform interface to all the supported technologies design objectives –easy to implement –enable efficient usage of middleware APIs close to existing middleware APIs –service-oriented –low-level –as many interfaces as ways to implement each functionality –optional interfacesplug-insinterfacesJSAGA core engine + plug-ins JSAGA SAGA SAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 7 Plug-ins interfaces: example Several way to monitor jobs –API mode  poll job status  listen to notifications about job status changes –API granularity  individual jobs  list of jobs  jobs filter (e.g. by user, by date, by tag…) Plug-in implements several interfaces User see a single interface: SAGA planned construction done Job monitoring SAGA user interface: getState / waitFor Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs query status for individual job listen status for individual job query status for filtered jobs gatekeepergLite-WMSwsgramunicore6 getState waitFor sshforkcreamnaregi core engine + plug-ins JSAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 8 Data Expression catalogrnslfnsrb / irodshttphttpssftprbyteiofilezipgsiftptarftpmailcachesrm basicdefaultJEPBeanShell Language JSDL+ext.SAGAJDLRSL-2RSL-4 Execution gatekeepergLite-WMSwsgramunicore6sshforkcreamnaregi Security InMemCredGlobusG. LegacyG. RFC820MyProxy VOMS Logical files core engine + plug-ins JSAGA JKSSSHLogin / pwdX509 Physical files Grid techno. Others Plug-ins (provided by and ) planned construction done

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 9 This is still not enough… job desc. gLite plug-ins Globus plug-ins JSAGA hidemiddlewareheterogeneity (e.g. gLite, Globus, Unicore) JDLRSL core engine + plug-ins JSAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 10 WMS WMS input data SRM GridFTP WS-GRAM LCG-CELCG-CEWS-GRAM firewall This is still not enough… job desc. gLite plug-ins Globus plug-ins JSAGA job staging graph delegate selection & files staging job hideinfrastructuresheterogeneity (e.g. EGEE, OSG, DEISA) hidemiddlewareheterogeneity (e.g. gLite, Globus, Unicore) JDLRSL OPlast EGEE

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 11 Ready-to-use software, adapted to targeted scientific fieldReady-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructuresHide heterogeneity between grid infrastructures Hide heterogeneity between middlewaresHide heterogeneity between middlewares As many interfaces as ways to implement each functionalityAs many interfaces as ways to implement each functionality As many interfaces as used technologiesAs many interfaces as used technologies core engine + plug-ins JSAGA jobscollection JSAGA end user application developer plug-ins developer SAGA SAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 12 Description of infrastructures Ready-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructuresHide heterogeneity between grid infrastructures Hide heterogeneity between middlewaresHide heterogeneity between middlewares World Grid EGEE OpenPlast localhost CC-IN2P3 srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS Globus Globus jobscollection JSAGA security context to use policy (e.g. firewalls) environment variables commands on workers (e.g. srmcp, wget…) heterogeneous APIs (e.g. WS-GRAM, WMS, Gatekeeper, SSH…)

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 13 url:// When using a single grid infrastructure –all files can be transported to/from the worker nodes through a single storage node When using several grid infrastructures –need to dynamically build a more complex transfer graph, according to… jobdesc.plug-insJSAGA jobscollection JSAGA World Grid EGEE OpenPlast localhost CC-IN2P3 srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS Globus Globus Transfer path depends on…

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 14 Transfer path depends on… grid or site –network filtering policy –commands available on workers –services available from workers (close Storage Element, shared FS) –supported context instances data to stage –shared by several jobs –installed on some worker nodes –file size –required data protection level execution service –protocols supported for staging transfer protocol –access mode (RO, WO, RW) –third-party transfer –supported data protection level jobdesc. World Grid EGEE OpenPlast localhost CC-IN2P3 srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS Globus Globus url://plug-insJSAGA jobscollection JSAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 15 jobscollection JSAGA OpenPlast OpenPlast World Grid EGEE OpenPlast localhost CC-IN2P3 srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS Globus Globus Transfer path depends on… grid or site –network filtering policy –commands available on workers –services available from workers (close Storage Element, shared FS) –supported context instances data to stage –shared by several jobs –installed on some worker nodes –file size –required data protection level execution service –protocols supported for staging transfer protocol –access mode (RO, WO, RW) –third-party transfer –supported data protection level OPlast GSIFTP EGEE CA SMTPGSIFTPGSIFTPHTTP OPlast SRB job C R1R1R1R1 E1E1E1E1 C' E1E1E1E1 R1R1R1R1 CC' common result std-error input files output files

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 16 jobscollection JSAGA OPlast EGEE CA SMTPGSIFTPGSIFTPHTTP GSIFTP OPlast SRB Transfer path depends on… grid or site –network filtering policy –commands available on workers –services available from workers (close Storage Element, shared FS) –supported context instances data to stage –shared by several jobs –installed on some worker nodes –file size –required data protection level job C"Esrc D1D1D1D1 E C R1R1R1R1 E1E1E1E1 C' TARTAR execution service –protocols supported for staging transfer protocol –access mode (RO, WO, RW) –third-party transfer –supported data protection level Sget E1E1E1E1 R1R1R1R1 CC' common result std-error input files output files EEsrc executable D1D1D1D1 input data

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 17 Ready-to-use software, adapted to targeted scientific fieldReady-to-use software, adapted to targeted scientific field Hide heterogeneity between grid infrastructuresHide heterogeneity between grid infrastructures Hide heterogeneity between middlewaresHide heterogeneity between middlewares As many interfaces as ways to implement each functionalityAs many interfaces as ways to implement each functionality As many interfaces as used technologiesAs many interfaces as used technologies Applications end user application developer plug-ins developer core engine + plug-ins JSAGA jobscollection JSAGA SAGA SAGA

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 18 Related projects JSAGA is used by…  a web portal for submitting jobs to industrial and research grid infrastructures –JJS (Java Job Submission)  a tool for submitting jobs to EGEE  optimized for short-life jobs (resource selection based on QoS observed while submitting jobs) –JUX (Java Universal eXplorer)  a multi-protocols file browser Applications / Applications Tested by to create security contexts

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 19 Related projects – JUX OS-independent –tested on Extensible –File content viewers  e.g. text, image, audio –Protocols and security  just copy JSAGA plug- ins in directory "lib/" Applications

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 20 Conclusion Main assets of JSAGA Implement standard specifications –SAGA –JSDL Provide high-level abstraction layer with no sacrifice on efficiency or scalability –thanks to design (definition of plug-ins interface) –thanks to cache mechanisms Use grid infrastructures as they are (i.e. no pre-requisite) –thanks to Hide heterogeneity middlewares –of middlewares grid infrastructures –of grid infrastructures World Grid EGEE OpenPlast localhost CC-IN2P3 tar:// srb:// gsiftp:// srm:// lfn:// gatekeeper gatekeeper wsgram WMS gatekeeper VOMS VOMS Globus

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 21 Perspectives Support new technologies –finish plug-ins in construction  CREAM  NAREGI (upgrade to v1.1)  SRM –develop new plug-ins  Gridway ?  French research grid middleware ?  … –integrate plug-ins developed by partners Implement new specifications –SAGA Extension: Service Discovery API  JSAGA currently has no equivalent for this  implementation will support plug-ins –JSDL Extension: Parameter Sweep Job  JSAGA currently does this in a non-standard way

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 22 Thank you !

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 23 Software quality Build process fully automated, including… –build tools installation –code generation –testing  unitary tests  integration tests –project web site generation  –installer GUI generation (see next slide…) Plug-ins –external dependencies reduced  e.g. gLite-UI not needed  most plug-ins supports –a maven 'archetype' generates skeleton of new plug-in project –plug-ins automatically validated with a reusable SAGA test suite # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base= https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 24 Installer GUI

Enabling Grids for E-sciencE EGEE-III INFSO-RI JSAGA 25 License(s) LGPL license –for the core engine and most plug-ins Optional licenses –for plug-ins having external dependencies, which license is not compatible with LGPL –then, end-user must…  either accept the terms of the license agreement  or uncheck these plug-ins (see previous slide)