Grid interoperability developments at CC-IN2P3 Lyon, 7 September 2010 Sylvain Reynaud

JSAGA2 Plan –the SAGA specification –the JSAGA implementation –example applications other grid interoperability developments at CC-IN2P3 –JJS –JUX –JPySAGA perspectives

JSAGA3 Simple API for Grid Applications SAGA is a specification –standard recommendation of –object oriented –simple 80/20 rule: 20% of the features cover 80% of the needs –integrated ("consistent") common "look & feel" for all the functional packages (see slide) –high-level hide low-level details about middlewares –middleware-independent use the same API to access all middlewares –language-independent Java C++ python (wrappers) –stable initial recommendation SAGA 1.0 (15 January 2008) a 1st update soon (backward compatible) –extensible Service Discovery, Message…

JSAGA4 Simple API for Grid Applications Source: S Jha, T Kielmann, A Merzky

JSAGA5 The JSAGA implementation JSAGA is an implementation of SAGA –written in/for Java focuses on –uniform usage of middleware…. (beyond what they natively support) –ease of extension………………. –efficiency and scalability………. –control of the API behavior (configuration rather than attempts) –operating-system independency (tested on ) Under LGPL license (thanks to the design of adaptor interfaces)

JSAGA6 Uniform usage of existing grids

JSAGA7 user applications VOMS WMS gsiftp Gatekeep. Globus JSAGA GridFTP Example – uniform usage SAGA job SAGA jobout.txtout.txt

JSAGA8 user applications VOMS WMS gsiftp Example – what happens behind EGEE proxy + attributes translate submit job poll status delegate transfer DGrid proxy adapt + translate submit job receive status transfer –to any protocol Gatekeep. Globus JSAGA CREAM-CELCG-CE Unicore WMS WMS Globus GK RSL VOMS VOMS modified job wrapper script GridFTP SAGA job D-Grid EGEE JDLout.txtout.txt

JSAGA9 Layered software architecture JSAGA interface implementation

JSAGA10 Layered software architecture Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces user applications middleware APIs core engine JSAGA adaptors JSAGA adaptors interfaces adaptors interfacesJSAGA SAGA SAGA Java binding interface implementation

JSAGA11 listen status changes transfer output Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces adaptor Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces

JSAGA12 Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces listen status changes transfer output

JSAGA13 Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces listen status changes Middleware API can be used more efficiently Middleware API can be used more efficiently –engine selects best interface for each user request Developing new adaptors is easier Developing new adaptors is easier –just wrap functionalities supported by middleware API transfer output

JSAGA14 LogicalPhysical Supported technologies Minimal dependencies on external libraries external tools –e.g. no gLite-UI operating system –tested on planned construction done Data Execution Security InMemCredGlobusG. LegacyG. RFC820MyProxyVOMSJKSSSHLogin / pwdX509gatekeepergLite-WMSwsgramunicore6creamnaregisshlocal rnslfnsrbirodscatalogrbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache Grid techno. Others

JSAGA15 Selection of security context Automatic selection is attractive to beginners, but it is… –inefficient waste of time while trying with wrong contexts jobs may fail at the end because they didn't run with expected VO/role –unreliable may create files that will not be accessible to partners may lock an account (e.g. by trying with 3 wrong passwords) –insecure sending all our passwords to unexpected services is not safe JSAGA's solution : configure selection of security context –bind job/data services to context types (+ URL patterns) –dynamic, per session, robust (all conflicts are detected at config-time )

JSAGA16 Example applications Pandora Gateway –the grid platform of the company a leading Grid/Cloud Computing Business Services Provider –used in several grid projects, including SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids used by experiences in –biology –computer science –physics –food industry

JSAGA17 Example applications jCAE (Java CAE) Computer Aided Engineering with support for distributed computing facilities web portal for industrial and academic production grids is used in several grid projects, including/ C alcul I ntensif pour les LO giciels de CAO E lectronique et les applications embarquées B io I nformatique (protein sequence alignement...) Plast urgy (simulation of injection…)

JSAGA18 Other interoperability tools – JJS Java Job Submission tool to submit jobs on EGEE/EGI grid infrastructure –efficient and scalable –resource selection based on the QoS observed while submitting jobs uses JSAGA in order to hide middleware evolutions for –security globus proxy  VOMS proxy –data management gridFTP  SRM –execution LCG-CE  CREAM-CE WMS used by D0 experiment to submit jobs/day on –EGEE/EGI (mainly Europe) –Open Science Grid (USA)

JSAGA19 Other interoperability tools – JUX Java Universal eXplorer multi-protocols file explorer extensible –for protocols just put the JSAGA plug-ins into the directory "lib/". –for viewers

JSAGA20 C Python Jython? JPySAGA Python Bindings for SAGA Java SAGAJSAGA Java GAT Java Bindings for SAGA Other interoperability tools – JPySAGA SAGA-C++ C++ Bindings for SAGA JySAGA a user application legacy python bind. Boost wrapper SAGA-C++Py a user application

JSAGA21 Upcoming… Develop new plug-ins –NorduGrid middleware in progress… –batch systems (DRMAA-based) Grid Engine –batch systems (CLI-based/ssh) torque Grid Engine Develop API extensions –Grid-RPC (from SAGA core) –Service Discovery API (a SAGA extension) with plug-ins for technologies –BDII with plug-ins for schemas –GLUE –CIM Interoperability demo at next OGF (Brussels, October) – –objective is not to show interoperability between middlewares, but to show interoperability between SAGA implementations

JSAGA22 Upcoming… Job submission –principle describe user's job submit it to the grid –middleware select the execution host –OGF standards BES, DRMAA, SAGA –examples Remote Procedure Call (RPC) –principle deploy user's service on grid call a procedure –middleware select the service instance –OGF standards Grid-RPC, SAGA –examples jobGrid-RPC

JSAGA23 Questions ?