1. Software Integration, Deployment & Evaluation
WP6: Integration Testbed
C. Loomis (LAL-CNRS)

2. Outline Status & Achievements Recommendations Objectives
Integration
Deployment
Evaluation & Feedback
Recommendations
Exploitation of Results
Questions
Objectives
Objectives of WP6 are best summarized by the Integration, Deployment, and Evaluation in the title. Must take middleware and external software, integrate it into a working release, deploy it on a geographically distributed testbed, and evaluate the software. Of course the results are fed back to developers.

3. Software Development Tools
Source Code Management (CVS)
184 developers
135 EDG Modules
108 MB of source
Autobuild
Package Repository
Testing Framework
Bugzilla
>2500 bugs
87% resolved
Pkg. Rep.
EDG
Pkgs. GB MB
EDG
43326
13.6
412
197
VDT
8915
2.1 12 44
Linux
4217
4.8
551
449
Extern.
874
1.0
104
204
Apps.
255
4.6
111
1498
Total
57587
26.1
1190
2392
The EDG SE has lots of small modules which boost this number. Still around 100 modules in the release.
To put the 108 MB of source code in perspective—the RH 7.3 linux kernel (another large open-source project) has 164 MB of code. The EDG project is of comparable complexity. (EDG has produced over 1 million lines of code.)
Autobuild system has been critical for generating consistent packages. Reduced time spent generating the packages and time wasted with trivial packaging problems.
Testing framework is shared between EDG and LCG. It contains a growing number of tests spanning site validation tests, basic functionality tests, and inter-service functionality tests.
Bugzilla has become the main forum by which problems with the software are funneled back to the developers. Detailed information about Bugzilla will be presented by Gabriel Zaquine in the quality report.

4. EDG Release Management
Integration Team
People from all work packages
Central to success of project
System Administrators
Efficient installation
Exemplary operation
Figure shows when production and development tags (vertical yellow bars) were released. Top two rows are production tags with the labels giving the last release of each series. (Production releases are expected to function coherently and be usable by applications.) Bottom row shows the test tags; these are not expected to be usable by applications.
Important points about release history:
LOTS of releases (64 production; 436 test tags)
Increased frequency because of installation tools and experience.
Highest frequency for 2.x series
Numerous new features
Simultaneous testing on app. & dev. Testbeds
Integration work continues!
High frequency possible because of WP4 installation and configuration tools. (And system administrators adoption and experience with them.) System administrators were generally happy with these tools (LCFGng) and are looking forward to using the next generation (quattor).
Many of the system administrators (especially those running central services) often went beyond the call of duty to keep services functioning for users. They often provided expert debugging and testing services also.

5. Application Testbed Total 161 14255 Physical Size of Testbed Users
Site
Country
CPUs
Storage (GB)
ASCC
Taiwan 21 8
Birmingham UK 4 34
Cambridge 5 16
CC-IN2P3 FR 3
348
CNAF IT 15
CPPM 2
ESRIN 28
FZK D
1034
Imperial Coll.
IFAE ES 88
IPSL
417
NIKHEF NL 32
1752
Oxford 35
Padova 71
Queen Mary Coll. 48
RAL 6 57
RAL-PP 10
SARA
10000
Sheffield
217
SRCE HR
UCL
Wuppertal
Total
161
14255
Physical Size of Testbed
21 sites in 8 countries
14.3 Terabytes
161 CPUs
Users
12 Virtual Organizations (VO)
100 regular users
500 registered users
Other projects
CrossGrid, DataTag, LCG
National Projects
EDG WP6 supported two testbeds:
Application Testbed for end-users
Development Testbed for development and testing
Also are various testbeds supported by middleware work packages for internal testing and development.
The application testbed is more limited in scope mainly because of competition for resources from other grid projects. This is actually positive in that site administrators trained on EDG software are taking their experience to other production grid projects.
These other projects are based on or use a significant part of the EDG software.
Some national projects include: GridPP, INFN Grid, NorduGrid.

6. Release Milestones EU Review (1.1.2) Features Prototype
Data-driven scheduling
Data mgt. services.
EU Review (1.1.2)

7. Release Milestones EU Review (1.4.3) Features
Good stability, reliability.
“Hybrid” information sys.
Push for automatic config.
On application testbed until released.
EU Review (1.4.3)

8. Release Milestones Evaluations (2.0.12) EU Review (2.1.13)
Features (2.0.12)
R-GMA replaces MDS
Refactored workload mgt.
Interactive, MPI, chkpt. jobs.
Replica Location Service
Webservice SE
Features (2.1.13)
Reasonable stability, reliability.
VOMS incorporated (WP1,4)
Bug fixes for all services.
There is a significant stabilization time for releases once they reach the application testbed. This typically lasts a few months. Have seen this both with the EDG 2.x series and EDG 1.4 series.
The integration, deployment, and testing were extremely active in fall of 2003 because EDG 2.0 series was on the application testbed with frequent R-GMA fixes deployed directly there. Simultaneous development of EDG 2.1 series on development testbed.

9. Security Infrastructure
Certificate Authorities
21 CAs span:
Europe
North America
Asia
Independent administration
Backbone of present & future global grid projects
VO Membership Service
Expect VO to take over operation servers
Include certificate requests into standard registration procedures
ArmeSFo
Armenia
HellasGrid
Greece
ASGCCA
Taiwan
INFN
Italy
CERN
Switzerland
LIP
Portugal
CESNET
Czech Rep.
NIKHEF
Netherlands
CNRS
France
NorduGrid
Nordic Countries
CyGrid
Cyprus
PolishGrid
Poland
ESNet
USA
Russia
FNAL
SlovakGrid
Slovakia
GermanGrid
Germany
Spain
GridCanada
Canada
UKeScience UK
Grid-Ireland
Ireland
The number of certificate authorities continues to expand. The CA coordination group will move under the auspices of EGEE. The overall coverage of users is significantly higher as CNRS CA signs requests for several countries and ASGCCA also covers China.
Details about security will be given by David Kelsey in the Security Report.

10. Support User Support System Administrator Support Users’ Guide
ITeam mailing list
Bugzilla
User tutorials
Service ticketing system exists
System Administrator Support
Installation Guide
ITeam/System Administrator mailing lists
System administrator tutorials
The service ticketing system is essentially unutilized. Perhaps it hasn’t been advertised widely enough, but CrossGrid (which shared system development) sees same result. More likely the fast, expert help available via the ITeam mailing list is more useful for the relatively “expert” users we now have.
Tutorial were a good technique in first part of project. Now most of the system administrators are really experts both with the installation tools and with the operation of the software.

11. Evaluation Private MW TBs Development TB Application TB
Formal Evaluation (D6.8)
Release procedures
Operations
EDG Middleware
Reality: Everyone Tests
Developers
Integrators
Testers
“Loose cannons”
End-users
Testing Suite
EDG development; shared with LCG.
Large (72) and growing body of tests
Private MW TBs
Development TB
Application TB
The formal evaluation used EDG release This was not a stable release and consequently the evaluation was very limited. The EDG 2.1 release is stable, but a formal evaluation has not been repeated.
D6.8 contains an evaluation of the release and operating procedures over the full life of the project. Many valuable recommendations for future grid projects are included in this document.
The developers typically do unit tests and isolated functionality tests on private testbeds. Integrators, testers and Loose Cannons work on candidate releases on development testbed. Loose Cannons and end-users test “production” releases on the application testbed.
The tests in the test suite are routinely used to vet releases and to check the configuration of sites. Having a common test suite helps catch problems early in the process.

12. Lessons Overall: Integration & Testing Effort Underestimated
Software Development
Service-oriented design
Code, packaging & configuration standards
Unified test framework
Operations
Auto-installation/config. tools mandatory
Minimize & decentralize infrastructure
Grid-enabled diagnostic tools
Fragmentation of Resources
Don’t have “webs” should not have “grids”.
Overall effort for integration and testing was underestimated. Many of the recommendations point towards solutions to make the integration easier.
Service-oriented design and the strict standards both will ease the integration of the software. This is important because one wants to be testing functionality of the release as a whole when integrating and not trivial problems with single components.
Having a common testing framework allows tests to be easily run at any level of the release procedure. Catching problems earlier is always better.
The installation tools may not be as critical in a production environment, but they are absolutely vital during the release building and validation processes.
With the grid being a collection of highly-correlated services, determining the cause of a given problem often requires examining multiple log file, sometimes on different sites. Diagnostic tools are necessary with allows one to do these correlations easily.
The balkanization of the global computing infrastructure is directly counter to the idea of a unified computing grid. Efforts needs to be made, particularly in the service discovery mechanisms to allow all resources to be accessible from anywhere.

13. Exploitation of Results
Software
Code & Support  partners, future projects
Infrastructure  limited, continued support
Package Rep./Web  frozen; available “forever”
Resources
App. TB  dev./demon. testbed for EGEE
CA Infrastructure  continues (coord. under EGEE)
VO Infrastructure  VOs must take over
Experienced People
Developers, integrators, etc.
System administrators
The assumption on the software services is that the bulk of the software will be taken over by EGEE. EGEE will then be responsible for running the software development services.
The resource owners themselves have ultimate control over where or if their resources are incorporated into another project. There are still some issues on how inclusive the EGEE fabrics will be.
The most valuable contribution to future grid projects are the people who have been involved in EDG. They take many recommendations from EDG to these projects, but also their personal expertise.