1. DGI: The D-Grid Infrastructure
Uwe Schwiegelshohn
University Dortmund
D-Grid Integration Projekt
Middleware

2. D-Grid Project Structure
6 separate community projects (funded for 3 years)
Astronomy (AstroGrid)
Climate research (C3-Grid)
Engineering (In-Grid)
High energy physics (HEP-Grid)
Medical research (MediGrid)
Humanities (TextGrid)
D-Grid integration project (DGI) (funded for 2 years)
4 areas of activity
Total funds: about 16 Mio €
Project start: September 2005

3. First D-Grid Extension
Projects of knowledge management and networking
Within the knowledge network
Linked flow of information in cooperative research
WIKINGER
Wiki Next Generation Enhanced Repository
WISENT
Knowledge network: Energy metereology
Ontoverse
Cooperative linked knowledge management within the area of life science
eSciDoc
A scientific platform for information, communication and publishing for research applications

4. Announced Second Extension
Additional communities
Involvement of industry
Horizontal activities
Bridging and linking several communities
Vertical activities
Entire new communities
Gap filling
Gaps determined during the D-Grid progress
Example: Cooperation between Shibboleth and VOMS
Total funds approx Mio € for 3 years

5. Community Grid Concept
Community focused Grid use
Top down approach
Driven by community researchers
Implementation of needed middleware services
Application oriented
Implementation by cooperating computer scientists
Approx. 30% of the total project effort
Detection of synergies
Cooperation between different communities
Middleware and tool transfer into the DGI
Testing, support, integration into existing middleware

6. Community Classification
Communities with significant Grid experience and involvement in international Grid projects
HEP-Grid, AstroGrid
Specific requirements in other communities
Security issues
MediGrid, In-Grid
Accounting and Billing
In-Grid
Large heterogeneous data files
C3-Grid
Semantic Grid and publication support
TextGrid

7. Project Risks
Separate community projects produce significant risks for D-Grid
Development process
Duplicate development efforts
Missing of synergy opportunities
Prolonged development time
Grid use
Insufficient support
User frustration
Interoperability
Use of non standardized middleware and tools
Barriers between different Grids

8. DGI Concept Support of the communities Focus on existing solutions
Tutorials and workshops
Installation packages
Counseling on use of tools
Focus on existing solutions
No major development effort in the first phase
Adaptation of available solutions to the needs of the communities
Migration to different platforms
Driven by community requirements
Detection of synergies
Integration of community middleware

9. DGI Areas of Activity Area 1 Area 2 Area 3 Area 4 Middleware
Data management
Virtual organizations
Area 2
Core D-Grid (exemplary D-Grid infrastructure)
Monitoring
Accounting and Billing
Area 3
Network
Security
Area 4
Coordination and dissemination

10. D-Grid Organization Structure
Steering Committee:
Project leader of each community project
Area coordinator of each DGI area
A neutral mediator with Grid experience
Goal:
Immediate detection of separation trends
Counteractive measures by the project leaders
Government intervention as a last resort
Start-up process:
Definition of architecture and requirements
Finding interaction partners on the technical level

11. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

12. Projects of DGI Area 1 Globus GT4 (basic services)
LCG/gLite (high level Grid services)
UNICORE (vertical middleware component)
GAT (mediation between application and services)
GridSphere (portal)
Data management
Data interfaces
Management of virtual organizations

13. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

14. Globus GT4: LRZ Munich Testing of GT4 components
GT 4 Migration to different platforms
AIX
Migration of applications running on GT2.x or GT3.x to GT4
Source Documentation (step-by-step guide)
Packaging of GT4 for D-Grid
Suse 9.3, 10.0, SLES 9 (LRZ)
RedHat 9, RedHat AS, Debian, Mac OS (Globus Team)
D-Grid support infrastructure for Globus
Cooperation with community projects
Monitoring (AstroGrid)
Close cooperation with the Globus Team (Argonne)

15. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

16. LCG/gLite: FZ Karlsruhe
Porting of EGEE middleware to not yet supported OS
Installation of gLite on various Sites:
RWTH Aachen, TU Dresden, University Dortmund, LMU Munich, MPI Munich (physics), Core D-Grid
Test accounts on Core D-Grid sites
HEP-Grid, AstroGrid, MediGrid
Installation packages managed by EGEE
gLite interoperability with
SUN Grid Engine as batch system.
AFS cluster file system
Packages transferred to EGEE
Cooperation with community projects
HEP-Grid
Involvement in EGEE

17. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

18. UNICORE: FZ Julich Interoperability with other middleware systems
Extension of GRIP components (UNICORE  GT2.4)
Development of interoperability component UNICORE gLite
Interoperability with GT4 via UNICORE/GS
UNICORE Atomic Services
Installation packages managed via
GUI for server installation packages (test phase)
Extension of UNICORE client configuration (development phase)
Cooperation with community projects
AstroGrid-D, specific functionalities (n-body UNICORE plugin)
D-Grid support infrastructure for UNICORE
UNICORE developer

19. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

20. GAT API: AEI Potsdam
Development or modification of GAT adapter components for other Grid services
PBS, SGE, UNICORE
Cooperation with community projects
AstroGrid (ProC Workflow Engine)
Maybe MediGrid, TextGrid (Python Wrapper)
D-Grid support infrastructure for GAT
Installation packages for D-Grid resources
GAT/ GAT2 (SAGA) developer

21. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

22. GridSphere: AEI, Fhg IAO
Installation packages for D-Grid resources
GridSphere Portal Framework and GridPortlets 1.2.1, available from the website
Cooperation with community projects
AstroGrid, In-Grid (intermediate results, StdOut/StdErr)
AstroGrid, MediGrid, In-Grid, HEP (login based on certificates)
Adaptation of Fraunhofer Resource Grid portal components to GridSphere (in progress)
D-Grid support infrastructure for GridSphere
GridSphere developer

23. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data interfaces
Data management
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

24. Data Management: DESY SRM/dCache SRB DataFinder OGSA/DAI
Installations RZG Munich, FZ Julich
SRB
Interoperability tests between GridFTP and SRB
Integration of data bases into SRB
Bug report to the developers (MCAT, DB Server)
Analysis of license and compatibility problems
DataFinder
Connection with OpenAFS (work in progress)
OGSA/DAI
Test installation am FZK with GT4.0.x
Analysis of security problems and available GUIs
Performance tests

25. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

26. Security: DFN and Others
Firewall concepts
Performance tests Cisco PX-70
Cooperative on demand opening (CODO) tests for UDP, TCP and SSL (work in progress)
AA infrastructure
Analysis of test installations with GT4.0, gLite (v.3.0), UNICORE
FZK and DFN are issuing Grid certificates.
VO management
HEP-Grid needs VOMS for compatibility reasons
C3-Grid and MediGrid want to use Shibboleth/Liberty
No available solution for cooperation between VOMS and Shibboleth or Liberty
Gap detection  development project for second extension
Grid specific CERT services
Cooperation with CERN/EGEE

27. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

28. Network: DFN and Others
Network extension with D-Grid specific elements
Network integration of T2/T3 centers
Alternative transport protocols
Evaluation of 10 GBit interfaces (IBM neterion, Intel PRO/10GbE CX4
Performance measurements with UTP, RDUDP and Tsunami and TCP variants

29. D-Grid Infrastructure
Community
User-/
application
interfaces
GridSphere
UNICORE
GAT API
Grid service
functions
D-Grid services
Nutzer
LCG/gLite
Accounting and Billing
Data management
Scheduling and Workflow Management
Scheduling and Workflow Management
Globus
Basic services
Monitoring
Security
Data interfaces
Available
D-Grid resources
Distributed
data archives
Data/
Software
Distributed
compute
resources
Network infrastructure

30. Scheduling and Workflow: HEP-Grid, C3-Grid
DGI does not directly address scheduling and workflow management
Partially included in gLite
Combined job scheduling and data management
Synergies in HEP-Grid and C3-Grid
Transfer and integration into D-Grid upon success

31. Grid Scheduling Architecture
Community
User-/ application- interface
Grid service
functions
Nutzer
Grid Scheduling
High Level Scheduling
High Level Scheduling
Basic services
Local resource
management
Scheduling
Scheduling
Scheduling
Available
Grid resources
Distributed
data archives
Data/ software
Distributed
compute resources
Network infrastructure

32. Conclusion
It is the goal of the D-Grid projects to generate a cooperative Grid infrastructure.
Inclusion of many different disciplines
Sustainable basic services and interfaces provided by the DGI participants
Generation of high level Grid functions within the community projects with the help of integrated computer scientists
Transfer of those Grid functions into the D-Grid infrastructure
Support of the community projects by the DGI project
Cooperation between the community projects
Integration of commercial applications and services