Status of Tsukuba, KEK, 21 September 2010 Sylvain Reynaud

JSAGA2 The SAGA specification SAGA : the specification –a recommendation of Open Grid Forum –a Simple API –for using Grid middleware execution management –job submission, monitoring... data management –transfer, list files, search… service discovery … –in user Applications implementations (or wrappers) exist for C++, Java and Python …in a uniform way S A G A use middlewares in a uniform way (independent of middlewares) use functionalities in a uniform way (consistent "look-and-feel") use language bindings in a similar way (language-independent specification)

JSAGA3 SAGA vs. gLite User Interface Source: S Jha, T Kielmann, A Merzky user gLite-UI VOMS, MyProx y WMS,LCG-CE,CREAM-CE SRM,gsiftpLFC, AMG A BDII

JSAGA4 Simple API for Grid Applications Java code example (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL(" gsiftp://cclcgseli01.in2p3.fr/tmp/ "); // create SAGA object with a factory NSDirectory dir = NSFactory.createNSDirectory(session, url); // use the created SAGA object List result = dir.list(); for (URL r : result) System.out.println(r); list content of directory gsiftp://…/tmp/

JSAGA5 Simple API for Grid Applications Java code example (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL(" wms://lapp-wms02.in2p3.fr/… "); // create SAGA object with a factory JobService svc = JobFactory.createJobService(session, url); // use the created SAGA object Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); submit a job and wait for the end of its execution

JSAGA6 The JSAGA implementation a Java implementation of the SAGA specification focuses on –uniform usage of middleware…. (beyond what they natively support) –ease of extension………………. –efficiency and scalability………. –control of the API behavior (configure binding rather than trying) –operating-system independency (tested on ) Under LGPL license (thanks to the design of adaptor interfaces)

JSAGA7 user applications VOMS WMS gsiftp Gatekeep. Globus JSAGA GridFTP Example – uniform usage SAGA job SAGA jobout.txtout.txt

JSAGA8 user applications VOMS WMS gsiftp Example – what happens behind EGEE proxy + attributes translate submit job poll status delegate transfer DGrid proxy adapt + translate submit job receive status transfer –to any protocol Gatekeep. Globus JSAGA CREAM-CELCG-CE Unicore WMS WMS Globus GK RSL VOMS VOMS modified job wrapper script GridFTP SAGA job D-Grid EGEE JDLout.txtout.txt

JSAGA9 Layered software architecture JSAGA interface implementation

JSAGA10 Layered software architecture Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces user applications middleware APIs core engine JSAGA adaptors JSAGA adaptors interfaces adaptors interfacesJSAGA SAGA SAGA Java binding interface implementation

JSAGA11 listen status changes transfer output Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces adaptor Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces

JSAGA12 Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces Used by end users Used by end users –web portals, GUI, CLI… Used by application developers Used by application developers –a single uniform interface –object oriented, high-level Implemented by adaptors Implemented by adaptors –each "way" to implement a given feature has 1 interface –service oriented, low-level Implemented by middleware Implemented by middleware –each technology has its own interfaces listen status changes transfer output

JSAGA13 Layered software architecture user applications GridFTP Globus GK adaptor get status get output upload data JSAGA core engine JSAGA adaptor interfaces listen status changes Middleware API can be used more efficiently Middleware API can be used more efficiently –engine selects best interface for each user request Developing new adaptors is easier Developing new adaptors is easier –just wrap functionalities supported by middleware API transfer output

JSAGA14adaptorsinterfacesJSAGA As many interfaces as ways to implement each functionality Example: ways to monitor jobs –API mode poll job status listen to notifications about job status changes –API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) planned construction done Job monitoring gatekeepergLite-WMSwsgramunicore6creamnaregi Design of adaptors interfaces sshlocal SAGA SAGA query status for individual job listen status for individual job query status for filtered jobs getState waitFor

JSAGA15adaptorsinterfacesJSAGA Design of adaptors interfaces Optional interfaces for optimization Example: ways to copy a file –data read/write methods either stream methods or get/put methods –data copy e.g. enable third-party transfer –delegated transfer planned construction done Physical rbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache SAGA SAGA data reader data writer data copy

JSAGA16 LogicalPhysical Supported technologies Minimal dependencies on external libraries external tools –e.g. no gLite-UI operating system –tested on planned construction done Data Execution Security InMemCredGlobusG. LegacyG. RFC820MyProxyVOMSJKSSSHLogin / pwdX509gatekeepergLite-WMSwsgramunicore6creamnaregisshlocal rnslfnsrbirodscatalogrbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache Grid techno. Others

JSAGA17 Latest developments New JSAGA adaptors –for data management gLite-LFC, dCache (SRM) –for execution management OGSA-BES (Unicore, ARC…) Improvements in some gLite adaptors and their libraries –features –thread-safety –memory leaks –… JPySAGA is now compliant with –reference Python binding for SAGA (as user interface) –reference Java binding for SAGA version 1.1 (as implementation interface) JSAGA has a new configuration engine –enable dynamic configuration –support 1 configuration/session –detect conflicts at config.-time

JSAGA18 Security context selection Try all security contexts attractive for beginners can be inefficient –waste of time (while trying) –waste of computing resource (risk of output staging failure) can be unreliable –may create files with wrong ownership –may block an account with too many failed connection attempts Bind contexts with URL patterns predictable/controllable behavior requires some knowledge about –used infrastructures –JSAGA configuration capabilities JSAGA has a new configuration engine –enable dynamic configuration –support 1 configuration/session –detect conflicts at config.-time

JSAGA19 Software quality Adaptors –validated by a middleware- independent SAGA test suite –maven 'archetype' to generate skeleton of new adaptor project Automated build process –download and install build tools external libraries –generate source code –execute test-suites unitary tests integration tests –generate project web site documentation reports –generate installer GUI extract library dependencies from maven project description # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base= https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/

JSAGA20 Installer GUI

JSAGA21 Licenses LGPL license –for the core engine and for most adaptors Optional licenses –for adaptors having external dependencies, which license is not compatible with LGPL –then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s)

JSAGA22 Example applications Pandora Gateway –the grid platform of the company a leading Grid/Cloud Computing Business Services Provider –used in several grid projects, including SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids used by experiences in –biology –computer science –physics –food industry

JSAGA23 Example applications jCAE (Java CAE) Computer Aided Engineering with support for distributed computing facilities web portal for industrial and academic production grids is used in several grid projects, including/ C alcul I ntensif pour les LO giciels de CAO E lectronique et les applications embarquées B io I nformatique (protein sequence alignement...) Plast urgy (simulation of injection…)

JSAGA24 Other interoperability tools – JJS Java Job Submission tool to submit jobs on EGEE/EGI grid infrastructure –efficient and scalable –resource selection based on the QoS observed while submitting jobs uses JSAGA in order to hide middleware evolutions for –security globus proxy  VOMS proxy –data management gridFTP  SRM –execution LCG-CE  CREAM-CE WMS used by D0 experiment to submit jobs/day on –EGEE/EGI (mainly Europe) –Open Science Grid (USA)

JSAGA25 Other interoperability tools – JUX Java Universal eXplorer multi-protocols file explorer extensible –for protocols just put the JSAGA plug-ins into the directory "lib/". –for viewers

JSAGA26 Example applications JSAGA provides command line interfaces for… –security jsaga-context-init jsaga-context-info jsaga-context-destroy –execution management jsaga-job-run jsaga-job-status jsaga-job-cancel jsaga-job-output Also useful as code examples to start with SAGA –data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical

JSAGA27 Interoperability of SAGA implementations Java SAGAJSAGASAGA-C++ Java Bindings for SAGAC++ Bindings SAGA several implementations of SAGA – –different languages (not necessarily) – –different characteristics features (e.g. automatic/configurable context selection, timeout…) optimizations (e.g. caching of connections, meta-data, states…) design choices (e.g. high-level / low-level adaptor interfaces) – –different supported API extensions (e.g. SD, Advert, Messaging) – –different supported technologies (adaptors) how to use all these API & technologies simultaneously ?

JSAGA28 Interoperability of Java implementations Java SAGAJSAGA Java Bindings for SAGA System.setProperty("saga.factory", IMPL_NAME); File f = FileFactory.createFile(url);// old code File f = FileFactory. getInstance(). createFile(url);// new code File f = FileFactory. getInstance(IMPL_NAME). createFile(url);// new (thread-safe) SAGA-C++ C++ Bindings SAGA Application + getInstance() Application

JSAGA29 Interoperability of Python implementations C Python Jython JPySAGA Python Bindings for SAGA (PySAGA) Java SAGAJSAGA Java Bindings for SAGA SAGA-C++ C++ Bindings SAGA JySAGA Application CppySAGA CppSAGA (legacy)

JSAGA30 Interoperability : all together… C Python Jython JPySAGA Python Bindings for SAGA (PySAGA) Java SAGAJSAGA Java Bindings for SAGA SAGA-C++ C++ Bindings SAGA JySAGA Application CppySAGA CppSAGA (legacy) + getInstance() SAGASAGA

JSAGA31 Perspectives Develop plug-ins for batch systems –DRMAA-based Grid Engine –CLI-based, with ssh torque Grid Engine Develop API extensions –Grid-RPC (from SAGA core) –Service Discovery API (a SAGA extension) with plug-ins for technologies –BDII with plug-ins for schemas –GLUE –CIM

JSAGA32 Perspectives Job submission –principle describe user's job submit it to the grid –middleware select the execution host –OGF standards BES, DRMAA, SAGA –examples Remote Procedure Call (RPC) –principle deploy user's service on grid call a procedure –middleware select the service instance –OGF standards Grid-RPC, SAGA –examples jobGrid-RPC

JSAGA33 Questions ?