DGI: The D-Grid Infrastructure Uwe Schwiegelshohn University Dortmund D-Grid Integration Projekt Middleware

D-Grid Project Structure 6 separate community projects (funded for 3 years) Astronomy (AstroGrid) Climate research (C3-Grid) Engineering (In-Grid) High energy physics (HEP-Grid) Medical research (MediGrid) Humanities (TextGrid) D-Grid integration project (DGI) (funded for 2 years) 4 areas of activity Total funds: about 16 Mio € Project start: September 2005

First D-Grid Extension Projects of knowledge management and networking Within the knowledge network Linked flow of information in cooperative research WIKINGER Wiki Next Generation Enhanced Repository WISENT Knowledge network: Energy metereology Ontoverse Cooperative linked knowledge management within the area of life science eSciDoc A scientific platform for information, communication and publishing for research applications

Announced Second Extension Additional communities Involvement of industry Horizontal activities Bridging and linking several communities Vertical activities Entire new communities Gap filling Gaps determined during the D-Grid progress Example: Cooperation between Shibboleth and VOMS Total funds approx. 15-20 Mio € for 3 years

Community Grid Concept Community focused Grid use Top down approach Driven by community researchers Implementation of needed middleware services Application oriented Implementation by cooperating computer scientists Approx. 30% of the total project effort Detection of synergies Cooperation between different communities Middleware and tool transfer into the DGI Testing, support, integration into existing middleware

Community Classification Communities with significant Grid experience and involvement in international Grid projects HEP-Grid, AstroGrid Specific requirements in other communities Security issues MediGrid, In-Grid Accounting and Billing In-Grid Large heterogeneous data files C3-Grid Semantic Grid and publication support TextGrid

Project Risks Separate community projects produce significant risks for D-Grid Development process Duplicate development efforts Missing of synergy opportunities Prolonged development time Grid use Insufficient support User frustration Interoperability Use of non standardized middleware and tools Barriers between different Grids

DGI Concept Support of the communities Focus on existing solutions Tutorials and workshops Installation packages Counseling on use of tools Focus on existing solutions No major development effort in the first phase Adaptation of available solutions to the needs of the communities Migration to different platforms Driven by community requirements Detection of synergies Integration of community middleware

DGI Areas of Activity Area 1 Area 2 Area 3 Area 4 Middleware Data management Virtual organizations Area 2 Core D-Grid (exemplary D-Grid infrastructure) Monitoring Accounting and Billing Area 3 Network Security Area 4 Coordination and dissemination

D-Grid Organization Structure Steering Committee: Project leader of each community project Area coordinator of each DGI area A neutral mediator with Grid experience Goal: Immediate detection of separation trends Counteractive measures by the project leaders Government intervention as a last resort Start-up process: Definition of architecture and requirements Finding interaction partners on the technical level

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Projects of DGI Area 1 Globus GT4 (basic services) LCG/gLite (high level Grid services) UNICORE (vertical middleware component) GAT (mediation between application and services) GridSphere (portal) Data management Data interfaces Management of virtual organizations

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Globus GT4: LRZ Munich Testing of GT4 components GT 4 Migration to different platforms AIX Migration of applications running on GT2.x or GT3.x to GT4 Source Documentation (step-by-step guide) Packaging of GT4 for D-Grid Suse 9.3, 10.0, SLES 9 (LRZ) RedHat 9, RedHat AS, Debian, Mac OS (Globus Team) D-Grid support infrastructure for Globus Cooperation with community projects Monitoring (AstroGrid) Close cooperation with the Globus Team (Argonne)

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

LCG/gLite: FZ Karlsruhe Porting of EGEE middleware to not yet supported OS Installation of gLite on various Sites: RWTH Aachen, TU Dresden, University Dortmund, LMU Munich, MPI Munich (physics), Core D-Grid Test accounts on Core D-Grid sites HEP-Grid, AstroGrid, MediGrid Installation packages managed by EGEE gLite interoperability with SUN Grid Engine as batch system. AFS cluster file system Packages transferred to EGEE Cooperation with community projects HEP-Grid Involvement in EGEE

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

UNICORE: FZ Julich Interoperability with other middleware systems Extension of GRIP components (UNICORE  GT2.4) Development of interoperability component UNICORE gLite Interoperability with GT4 via UNICORE/GS UNICORE Atomic Services Installation packages managed via UNICORE@SourceForge GUI for server installation packages (test phase) Extension of UNICORE client configuration (development phase) Cooperation with community projects AstroGrid-D, specific functionalities (n-body UNICORE plugin) D-Grid support infrastructure for UNICORE UNICORE developer

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

GAT API: AEI Potsdam Development or modification of GAT adapter components for other Grid services PBS, SGE, UNICORE Cooperation with community projects AstroGrid (ProC Workflow Engine) Maybe MediGrid, TextGrid (Python Wrapper) D-Grid support infrastructure for GAT Installation packages for D-Grid resources GAT/ GAT2 (SAGA) developer

D-Grid Infrastructure Community User-/ application interfaces GridSphere GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

GridSphere: AEI, Fhg IAO Installation packages for D-Grid resources GridSphere Portal Framework 2.1.2 and GridPortlets 1.2.1, available from the website Cooperation with community projects AstroGrid, In-Grid (intermediate results, StdOut/StdErr) AstroGrid, MediGrid, In-Grid, HEP (login based on certificates) Adaptation of Fraunhofer Resource Grid portal components to GridSphere (in progress) D-Grid support infrastructure for GridSphere GridSphere developer

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data interfaces Data management Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Data Management: DESY SRM/dCache SRB DataFinder OGSA/DAI Installations RZG Munich, FZ Julich SRB Interoperability tests between GridFTP and SRB Integration of data bases into SRB Bug report to the developers (MCAT, DB Server) Analysis of license and compatibility problems DataFinder Connection with OpenAFS (work in progress) OGSA/DAI Test installation am FZK with GT4.0.x Analysis of security problems and available GUIs Performance tests

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Security: DFN and Others Firewall concepts Performance tests Cisco PX-70 Cooperative on demand opening (CODO) tests for UDP, TCP and SSL (work in progress) AA infrastructure Analysis of test installations with GT4.0, gLite (v.3.0), UNICORE FZK and DFN are issuing Grid certificates. VO management HEP-Grid needs VOMS for compatibility reasons C3-Grid and MediGrid want to use Shibboleth/Liberty No available solution for cooperation between VOMS and Shibboleth or Liberty Gap detection  development project for second extension Grid specific CERT services Cooperation with CERN/EGEE

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Network: DFN and Others Network extension with D-Grid specific elements Network integration of T2/T3 centers Alternative transport protocols Evaluation of 10 GBit interfaces (IBM neterion, Intel PRO/10GbE CX4 Performance measurements with UTP, RDUDP and Tsunami and TCP variants

D-Grid Infrastructure Community User-/ application interfaces GridSphere UNICORE GAT API Grid service functions D-Grid services Nutzer LCG/gLite Accounting and Billing Data management Scheduling and Workflow Management Scheduling and Workflow Management Globus Basic services Monitoring Security Data interfaces Available D-Grid resources Distributed data archives Data/ Software Distributed compute resources Network infrastructure

Scheduling and Workflow: HEP-Grid, C3-Grid DGI does not directly address scheduling and workflow management Partially included in gLite Combined job scheduling and data management Synergies in HEP-Grid and C3-Grid Transfer and integration into D-Grid upon success

Grid Scheduling Architecture Community User-/ application- interface Grid service functions Nutzer Grid Scheduling High Level Scheduling High Level Scheduling Basic services Local resource management Scheduling Scheduling Scheduling Available Grid resources Distributed data archives Data/ software Distributed compute resources Network infrastructure

Conclusion It is the goal of the D-Grid projects to generate a cooperative Grid infrastructure. Inclusion of many different disciplines Sustainable basic services and interfaces provided by the DGI participants Generation of high level Grid functions within the community projects with the help of integrated computer scientists Transfer of those Grid functions into the D-Grid infrastructure Support of the community projects by the DGI project Cooperation between the community projects Integration of commercial applications and services