FP6−2004−Infrastructures−6-SSA Interoperability Task in EUChinaGrid Project Giuseppe Andronico INFN Sez. Di Catania OGF 20 - GIN Manchester,

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  2/20  Agenda Overview of our work in interoperability Gateway-based interoperability used in the project Role of Gateway Gateway design principles Core components of gateway Batch job level interoperability process  submit batch job from GOS to GLite  submit batch job from GLite to GOS How to deal with Data transfer  Manner to implement data transfer in EUChinaGRID project Security issues What to do next

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  3/20  Overview of our work in interoperability Up to now our work mainly covers the following three aspects: Design flexible gateway and want to propose generic design for similar scenarios  Use SEDA model as task process tool  Use IoC model as configuration and assembly tool Extend CNGrid GOS JobManager Framework Extend GLite LCG-CE JobManager Framework What we have achieved? Finished and deployed our first implementation in testbeds we set up in IHEP (CAS) and in Catania (INFN)  Up to now these two testbeds have ran stably for nearly three months  Process more than 1,500 batch jobs in total [ including both GOS to GLite and GLite to GOS] Passed the first EU project review which was held in February 27th in Madrid

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  4/20  Agenda Overview of our work Gateway-based interoperability used in the project Role of Gateway Gateway design principles Core components of gateway Testbed Batch job level interoperability process  submit batch job from GOS to GLite  submit batch job from GLite to GOS How to deal with Data transfer  Manner to implement data transfer in EUChinaGrid project Security issues What to do next in EUChinaGrid project

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  5/20  Role of Gateway Gateway is just a logical component, it can be handled as a “facade” of underlying grid infrastructure: Interface conversion Function mapping etc Gateway should support the following features: Transparent to end users of different grid infrastructures Easy to extend Support massive concurrency and high throughput Support standalone deployment or integrated underlying grid middleware etc

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  6/20  Gateway design principle Gateway design depends heavily on SEDA model and IoC model The reasons to choose SEDA model and IoC model: SEDA model  SEDA is short for Staged Event Driven Architecture  Firstly proposed by Matt Welsh, David Culler, and Eric Brewer, from University of California, Berkeley in their paper SEDA: An Architecture for Well-Conditioned, Scalable Internet Services  Support massive concurrency, high throughput  Simplify the construction of well-conditioned Internet services  In our design, we firstly divide the whole process into several independent basic stages and then compose basic stages into different pipelines for different purpose such as gLite-to-GOS batch job forwarding and so on IoC model  IoC is short for Inversion of Control  Provide loose couple among different modules and easy to reuse basic modules  Assemble new module easily and quickly  In our design, HiveMind 1.1 is used as IoC container which is released under LGPL license [It means we can use it freely in our project]

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  7/20  Core components of Gateway Core components of gateway including: Pipelines constructed for different purposes  Consists of different basic processing stages  Used for different purposes such as forwarding batch job from GOS to gLite and vice versa Scheduler  Executes processing stages at fixed rate  One to one mapping between Pipeline and scheduler Thread pool  Improves performance  One to one mapping between Thread pool and scheduler Different processing stages in same pipeline usually in charge of different concrete functions such as StageIn, StageOut and so on Detailed description of gateway can be found in the next slide

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  8/20  GOS WMProxy batch job Extended LCG-CE batch job GOS batch job Thread Pool scheduler Thread Pool scheduler Detailed description of gateway components Pipeline for GLite to GOS Pipeline for GOS to GLite 1 Different colors in pipeline are for different stages which are in charge of concrete functions such as data stageIn, data stageOut and so on 2 Different pipelines have different thread pool and scheduler Extended LCG-CE forward batch job to gateway Extended GOS forward batch job to gateway idle threads pool used in schedule Scheduler execute stage in pipeline using idle thread from thread pool

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  9/20  Batch job level interoperability process To implement batch job level interoperability, we did the following work: Extended JobManager in both gLite and GOS  Extended gLite LCG-CE JobManager Framework  LCG-CE JobManager Framework has couple relationship with resource schedule mechanism of gLite  Relatively difficult, cost a lot of time  Provided Broker plugin for GOS JobManager framework Sandbox mode data transfer A fast approach for cross-domain security scenario A detailed introduction will be found in the next two slides

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  10/20  Testbed in Catania, INFN portal.ct.infn.it gos.ct.infn.it glite-gos.ct.infn.it WMProxy glite-rb2.ct.infn.it OpenPBS PipeLine4GLite PipeLine4GOS WMS/RB GLite-UI Extended LCG-CE Gateway component Portal or WS-Client Command Line CE WN JSDL JDL RSL JDL RSL JSDL

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  11/20  Batch job level interoperability process GOS Node PipeLineForGLite WMProxy JSDL Forwarding JSDL Submit JDL Extended LCG-CE Gateway Component glite-gos.ct.infn.it portal.ct.infn.it Dispatch batch job WMS PipeLineForGOS Submit JSDL Submit JDL GLite-UI Submit JDL glite-rb2.ct.infn.it PipeLineForGLite: 1 Convert JSDL to JDL 2 Data Transfer 3 Submit job to WMProxy Extended LCG-CE: 1 Extend Globus JobManager used in LCG-CE 2 Convert RSL to JSDL 3 Submit batch job to PipeLineForGOS PipeLineForGOS: 1 Data Transfer 2 Submit job to GOS GOS Node gos.ct.infn.it

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  12/20  Data Transfer Data transfer operations between CNGrid and EGEE uses sandbox mode All the data transfer will pass the gateway Suitable for small scale data transfer scenario Gateway act as data transfer center and have two different roles at the same time:  GridFTP client  Gateway upload/download necessary data to/from gLite WMS  FTP server  GOS upload/download data to/from gateway component

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  13/20  Role of gateway component in Data Transfer GridFTP Client FTP Server GridFTP ServerFTP Client GridFTP Protocol FTP Protocol FTP protocol is widely used in CNGrid environment, so we used ftp protocol to transfer data between GOS node and gateway GridFTP protocol is widely used in glida environment, so we used ftp protocol to transfer data between GLite RB and gateway Gateway Component

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  14/20  Security issues Security module used now is just a fast approach for cross-domain security scenario Predefine some users in GOS and gLite for interoperability purpose, just a static approach  Request from GOS are submitted to gLite using predefined voms proxy  Requests from gLite are submitted to GOS using predefined name User Management module is designed to keep mapping relationships Security token service Used to keep, distribute, exchange and verify security token between GOS and gLite and provide dynamical approach At present MyProxy Server is setup to store temporary security token, but we plan to replace it with newly developed security token service

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  15/20  Role of security token service GOS GLite Gateway Security token Service security token still obey the way that GOS uses for service invocation still obey the way that GLite uses for service invocation WS-Trust Client Role of security token service in cross-domain security scenario

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  16/20  Agenda Overview of our work Gateway-based interoperability used in the Project Role of Gateway Gateway design principles Core components of gateway Testbed Batch job level interoperability process  submit batch job from GOS to GLite  submit batch job from GLite to GOS How to deal with Data transfer  Manner to implement data transfer in EUChinaGrid project Security issues What to do next in EUChinaGrid Project

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  17/20  What to do next in EUChinaGrid Project Before the end of the EUChinaGrid project, we plan to work on the following fields: Cross-domain Security  Security token service based token distribution in cross- domain scenario:  More generic solution for cross-domain security token distribution  Comply with WS-Trust specification Large scale data share and transfer between different grid infrastructures  At present we make full use of sandbox mode to support data transfer between GOS and gLite and all the data transfered between GOS and gLite pass the gateway. The problems lie in:  Do unnecessary data transfer between the two middlewares  Cause problems in the case of large scale data transfer

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  18/20  What to do next in EUChinaGrid Project Comply with some work of OGF GIN Group  OGSA-BES  SRM Support real grid application interoperability between CNGrid and EGEE  POSIX application is supported now  Choose from applications supported by EUChinaGrid project

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  19/20  Reference materials During preparing for this presentation, I use materials from following paper or presentations and thanks a lot the authors SEDA: An Architecture for Well-Conditioned, Scalable Internet Services, Matt Welsh, David Culler, and Eric Brewer, University of California, Berkeley Distributed Computation, technical report of London e- Science Centre (Le-Sc), Imperial College, London State-of-the-art of Interoperability in EUChinaGrid Project, EUChinaGrid Project Annual Work Report HiveMind,

Giuseppe Andronico  INFN Sez. CT  OGF 20 GIN  Beijing,  20/20  Thanks for your attentions Any Questions?