1. IGTMD meeting, Lyon Sylvain Reynaud
Current status of
IGTMD meeting, Lyon
Sylvain Reynaud

2. Use cases User may access several grid infrastructures when he needs…
some data available on a given grid
a very large number of computing resources
resources with specific constraints
super-computer
confidentiality
small overhead (e.g. for consolidation)
interactivity …
cluster
JSAGA

3. (e.g. gLite, Globus, Unicore)
What is JSAGA ?
job
desc.
JDL
RSL
JSAGA
(e.g. gLite, Globus, Unicore)
middleware
heterogeneity
hide
gLite
plug-ins
Globus
plug-ins
JSAGA

4. (e.g. gLite, Globus, Unicore)
What is JSAGA ?
job
desc.
EGEE
OPlast
JSAGA
(e.g. gLite, Globus, Unicore)
middleware
heterogeneity
hide
gLite
plug-ins
Globus
plug-ins
JDL
RSL
staging
graph
delegate selection
& files staging
job
job
WMS
input
data
SRM
GridFTP
WS-GRAM
LCG-CE
firewall
infrastructures
heterogeneity
(e.g. EGEE, OSG, DEISA)
hide grid
JSAGA

5. Global architecture
For end-users: ready-to-use software adapted to their needs
For application developers: hide infrastructure heterogeneity
For application developers: hide middleware heterogeneity
For plug-in developers: as many interfaces as ways to implement the functionalities
Class URLFactory introduced in release "1.0rc2".
This will enable to implement :
support for windows paths
support for relative paths
protocol-dependant URL encoding …
SAGA
Plug-ins interfaces
JSAGA

6. Design of plug-in interfaces
close to application developer needs
object oriented
uniform interface to all the supported technologies
but << certainly not simple to implement >> (T. Kielmann)
code is twice bigger than the code for all the plug-ins
close to existing middleware API
service oriented
as many interfaces as ways to provide the functionalities
optional interfaces for technology-specific…
optimizations
additional features
SAGA
Plug-ins interfaces
Need standard plug-in interfaces to enable to
factorize efforts of wrapping middleware API.
JSAGA

7. Plug-in interfaces for execution
Streaming
Plug-in interfaces:
direct/buffered/redirected streams
used before/during/after execution
Monitoring
Plug-in interfaces:
querying / listening
individual job / list of jobs / filtered jobs
set stream
for
interactive
set stream
for non-
interactive
get stream
for
interactive
query
status for
individual
job
listen
status for
individual
job
query
status for
filtered
jobs
getInput
getOutput
getError
getState
waitFor
SAGA user interface:
getInput / getOutput
SAGA user interface:
getState / waitFor
Job control
Job monitoring
gatekeeper
gLite-WMS
wsgram
unicore6
ssh
fork
cream
gridvm
PBS
proxy
gatekeeper
gLite-LB
wsgram
unicore6
ssh
fork
cream …
planned
construction
done
JSAGA

8. Provided plug-ins Job control Security Data Exec. Exec. Expression
Physical files
Logical files
X509
SSH
InMemCred
Globus
G. Legacy
G. RFC820
MyProxy
VOMS
Login / pwd
JKS
Data
catalog
rns
lfn
srb / irods
http
https
sftp
rbyteio
file
zip
gsiftp
tar
ftp
mail
cache
srm
Exec.
(control)
Exec.
(monitor)
Job control
gatekeeper
gLite-WMS
wsgram
unicore6
ssh
local
cream
gridvm
PBS
remote
gatekeeper
gLite-LB
wsgram
unicore6
ssh
local
cream …
Expression
Language
basic
default
JEP
BeanShell
JSDL+ext.
SAGA
JDL
RSL-2
RSL-4
planned
construction
done
JSAGA

9. Plug-ins out of SAGA scope
piece of a sample JSDL job
description in
JDL
description in
JSDL
description in
RSL
<POSIXApplication>
<Executable>myexec</Executable>
now+24*INDEX)}</Argument>
</POSIXApplication> …
Job Collection
Job
JSDL
retry
rules
termination
parametric
language
specific
data staging
extensions
<Executable>myexec</Executable>
<Argument>-from 30/05/2008</Argument>
<POSIXApplication>
</POSIXApplication> …
<Executable>myexec</Executable>
<Argument>-from 31/05/2008</Argument>
<POSIXApplication>
</POSIXApplication> …
<Executable>myexec</Executable>
<Argument>-from 01/06/2008</Argument>
<POSIXApplication>
</POSIXApplication> …
Expression
Language
basic
default
JEP
BeanShell
JSDL+ext.
SAGA
JDL
RSL-2
RSL-4
planned
construction
done
JSAGA

10. (e.g. gLite, Globus, Unicore)
Demo (part 1)
hide
middleware
heterogeneity
(e.g. gLite, Globus, Unicore)
Using SAGA API
security context
data management
execution management
Gatekeeper
Unicore 6
SSH
fork
export PATH="$PATH:$PWD/bin"
jsaga-context-init.bat Globus
jsaga-context-info.bat
jsaga-ls.bat file://c:/
jsaga-ls.bat
jsaga-cat.bat
cd doc/examples/test_demo1/
jsaga-job-run.bat -Executable f:/cygwin/bin/date.exe -Arguments --utc –r <resource>
JSAGA

11. Global architecture
For end-users: ready-to-use software adapted to their needs
For application developers: hide infrastructure heterogeneity
For application developers: hide middleware heterogeneity
For plug-in developers: as many interfaces as ways to implement the functionalities
Job Collection
SAGA
Plug-ins interfaces
JSAGA

12. Describe grid infrastructures
requirements
VOMS
Hide middleware heterogeneity
Uniform interface (SAGA)
Hide infrastructure heterogeneity
Selection of the right security context
Selection of the right client API configuration
Transport of the job input/output data to/from worker nodes
Describe the capabilities of the infrastructures you want to use
CC-IN2P3
WMS
gsiftp://
EGEE
VOMS
wsgram
OpenPlast
Grid
Globus
localhost
World
JSAGA

13. Describe grid infrastructures
examples
VOMS
gatekeeper
Hide middleware heterogeneity
Job management technologies
e.g. CREAM, WMS, SSH, GK
Hide infrastructure heterogeneity
Computing Elements (e.g. GK)
Different grid or site policies
e.g. network filtering, shared FS
Different environment variables
e.g. $VO_?_SW_DIR, /usr/local
Different configuration attributes
e.g. monitor service URL, shell path on cygwin, default SE URL
Commands available on worker
e.g. globus-url-copy, srmcp, Scp, wget, tar
srb://
srm://
CC-IN2P3
WMS
lfn://
gsiftp://
EGEE
VOMS
gatekeeper
wsgram
OpenPlast
Grid
Globus
tar://
gatekeeper
localhost
World
JSAGA

14. Transfer path depends on…
World
Grid
EGEE
OpenPlast
localhost
CC-IN2P3
tar://
srb://
gsiftp://
srm://
lfn://
gatekeeper
wsgram
WMS
VOMS
Globus
Using a single grid
all files can be transferred to the worker through a single node
Using several grids
need to dynamically build a transfer graph, according to…
url://
job
desc.
JSAGA
plug-ins
JSAGA

15. Transfer path depends on…
grid or site
network filtering policy
commands available on workers
services available from workers (close Storage Element, shared FS)
supported context instances
execution service
protocols supported for staging
transfer protocol
access mode (RO, WO, RW)
third-party transfer
supported data protection level
data to stage
shared by several jobs
installed on some worker nodes
file size
required data protection level
World
Grid
EGEE
OpenPlast
localhost
CC-IN2P3
tar://
srb://
gsiftp://
srm://
lfn://
gatekeeper
wsgram
WMS
VOMS
Globus
url://
job
desc.
JSAGA
plug-ins
JSAGA

16. Transfer path depends on…C C'
common
result
std-error
Transfer path depends on…
grid or site
network filtering policy
commands available on workers
services available from workers (close Storage Element, shared FS)
supported context instances
execution service
protocols supported for staging
transfer protocol
access mode (RO, WO, RW)
third-party transfer
supported data protection level
data to stage
shared by several jobs
installed on some worker nodes
file size
required data protection level
OPlast
EGEE CA
SMTP
GSIFTP
HTTP
SRB C R1 E1 C'
OpenPlast
OPlast
GSIFTP
job
World
Grid
EGEE
OpenPlast
localhost
CC-IN2P3
tar://
srb://
gsiftp://
srm://
lfn://
gatekeeper
wsgram
WMS
VOMS
Globus
OPlast
OpenPlast
JSAGA

17. Transfer path depends on…C C'
C''
common
Transfer path depends on… E
src
executable D1
input data R1
result
grid or site
network filtering policy
commands available on workers
services available from workers (close Storage Element, shared FS)
supported context instances
execution service
protocols supported for staging
transfer protocol
access mode (RO, WO, RW)
third-party transfer
supported data protection level
data to stage
shared by several jobs
installed on some worker nodes
file size
required data protection level E1
std-error
OPlast
EGEE CA
SMTP
GSIFTP
HTTP
SRB C R1 E1 C' D1 E
src
C"
OPlast
job
OPlast E
TAR
TAR
iGet
JSAGA

18. (e.g. gLite, Globus, Unicore)
Demo (part 2)
hide
middleware
heterogeneity
(e.g. gLite, Globus, Unicore)
hide grid
infrastructures
heterogeneity
(e.g. EGEE, OSG, DEISA)
Using SAGA API
security context
data management
execution management
Gatekeeper
Unicore 6
SSH
fork
Using Job Collection API
execution mgt.
EGEE
OpenPlast
my laptop
World
Grid
EGEE
OpenPlast
localhost
CC-IN2P3
tar://
srb://
gsiftp://
srm://
lfn://
gatekeeper
wsgram
WMS
VOMS
Globus
vi etc/jsaga-engine.properties
more doc/examples/jsaga-config.xml
jsaga-help.bat --config | more
cd doc/examples/test_demo2/
more job.xml
jsaga-jobcollection-run.bat job.xml -r resources.xml –dump-wrapper MYJOB_2 | more
jsaga-jobcollection-run.bat job.xml -r resources.xml --graph --name all
# name openplast_sbx
# name openplast
# name egee
JSAGA

19. Software quality Fully automated build process
download build tools, modules
generate code, build, run tests
Generated project web site :
documentation
user guides (from annotations)
specification (from code)
design (from code)
reports
static code analysis
test results and coverage
tasks list
Provide wizard to generate skeleton of new plug-in projects
Provide reusable test suite for validating plug-ins
# SAGA protocols test-suite configuration
gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/
gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/
gsiftp.context=OpenPlast_proxy
lfn.base=lfn:///grid/dteam/
lfn.context=EGEE_proxy
http.base=
file.base=file:///c:/tmp/
file.base2=file:///c:/
JSAGA

20. Software quality Minimize dependencies Provide an installer
resolve conflicts between versions of libraries
with maven
regenerate stubs from WSDL
with Axis 1.4
remove external dependencies
no gLite-UI for EGEE, no OS-specific dependency
Provide an installer
install only what you need…
JSAGA

21. Software metrics Specification Implementation
SAGA, version 1.0 (January 15, 2008) : 323 pages
Official Java Binding, version "snapshot-1"
Implementation
46 third-party libraries (only 5 are for the core engine)
644 java classes (+ 508 generated java classes)
some code in other languages C
XSL
XML Schema
LL grammar
Bash shell
MS-DOS batch
JSAGA

22. Main assets of JSAGA
does not sacrifice scalability and efficiency for abstraction
thanks to design (in particular the definition of plug-in interfaces)
thanks to caching mechanisms
does not only hide heterogeneity of middleware, but also hides heterogeneity of grid infrastructures
thanks to
use existing grid infrastructures as they are
World
Grid
EGEE
OpenPlast
localhost
CC-IN2P3
tar://
srb://
gsiftp://
srm://
lfn://
gatekeeper
wsgram
WMS
VOMS
Globus
plug-ins
JSAGA
url://
job
desc.
JSAGA

23. Global architecture
For end-users: ready-to-use software adapted to their needs
For application developers: hide infrastructure heterogeneity
For application developers: hide middleware heterogeneity
For plug-in developers: as many interfaces as ways to implement the functionalities
Applications
Job Collection
SAGA
Plug-ins interfaces
JSAGA

24. / Related projects JSAGA is used by… Applications Elis@
a web portal for submitting jobs to industrial and research grid infrastructures
JJS (Java Job Submission)
a tool for submitting job collections to EGEE
optimized for short-life jobs (resource selection based on QoS observed while submitting jobs)
JUX (Java Universal eXplorer)
a multi-protocol file browser
is currently testing it /
JSAGA

25. Perspectives Improvement of the core engine (IGTMD project)
move to SAGA Java Binding release candidate 2 (and next...)
bug-fixes, new features, etc.
Development of plug-ins (contribution to other projects)
RA-Grid
plug-ins for OAR, CiGri
NAREGI
plug-ins for Super Scheduler, (GridVM ?)
for BQS users who want to progressively use grid
plug-ins for BQS
JJS
plug-ins for CREAM, LCG-CE (extends GK plug-in),
finish plug-ins for SRM, LFC
plug-ins for JDL, tar
etc.
JSAGA