1. LCG-1 Status & Task Forces
Ian Bird
LCG & IT Division, CERN
GDB Meeting
10 June 2003

2. Task Forces Security Grid Operations Centres Call Centre
Monthly meetings – 2 weekly phone conferences – very active
See Dave Kelsey’s talk
Grid Operations Centres
See Trevor Daniels’ talk for longer term – would like a group to address this
See John Gordon’s talk for short term
Call Centre
See Klaus-Peter Mickel’s talk
Grid File Access library
Being tested against Enstore/dCache (FNAL)
Prototype implementation for July
Deployment organisation
See below
Middleware integration, testing, packaging
Storage interfaces –
Propose a JTB project to agree on inter-operating SRM/SE implementations
Site integration issues:
Local
TCP connectivity to worker nodes

3. LCG-1 (July) Middleware
Basic LCG service in July will be based on:
VDT 1.1.8,
Update including our bug fixes, etc ready
RLS
Basic service with single LRC, data management tools
We have already tested this stand-alone, shown to be better than old EDG replica catalog
Resource Broker
Many changes – to improve scalability issues, but still to be tested
Information system
MDS:
Some bug fixes, use static config files if needed
R-GMA:
Still problems – will be deployed if/when possible – needs testing and real comparison with MDS
Make real comparison in certification testbed deployed system
Our work on MDS will provide baseline for comparison (see below)

4. Actual status Tagged version from EDG 2.0
Installs and services start
RLS runs with both MDS and R-GMA
RB not yet working
R-GMA more or less stable, but not seriously tested yet (no RB job submission)
No real testing has yet been done!
Testing and bug fixing will be our top priority
Starting as soon as possible (this week?)
In parallel with preparing the deployment

5. MDS Stabilisation The MDS problems Problems observed:
MDS 2 as distributed by vdt1.1.8.
MDS is operated by building a hierarchy. The local GRIS on a node registers with a site GIIS which in turn registers with a next level of the hierarchy (country to top level).
The top level MDS is used as an II (information Index).
The II contains the information as well as index to it.
Problems observed:
If multiple sites (>5) connect to a higher level GIIS the GIIS becomes unavailable for long periods. This has a bad effect on the performance of and resource management since the discovery of resources is stalled until the GIIS runs again.
The current fixes (EDG):
Parallel to the top level MDS node the BDII has been introduced. This is a simple LDAP server that samples with the help of a cron job the top level MDS and offers this information via LDAP.
This improved the availability of the service, but data can be stale. If a resource (CE) is gone, but still reported by the BDII the resource broker (RB) will query the GRIS on this CE and wait for the timeout
Since each queue appears as an independent CE a single broken CE can make the matchmaking take an very long time.
What we have discovered:
If a GRIS that has registered with a GIIS goes down in a non-graceful way the GIIS doesn't respond for several minutes (about 3). There seems to be a parameter in the Nordu-Grid version to configure this timeout.
The simple GRIS has been known to be quite reliable.

6. Our current plan:
Replace simple hierarchy with one with more than one top on the level above the site level GIIS
The GRIS registers with a site level GIIS which in turn registers with two or more regional GIIS.
The contact information for these are kept in a semi-static list (i.e maintained by hand) to configure the next level of the information systems.
To configure the site GIIS it is only necessary to know in which region you are.
By shrinking the size of a region these intermediate servers can be adjusted to the amount of information to be provided and
By defining the number of replications a higher level of overall availability and sharing of the query load can be achieved.
The top level MDS will be replaced by a BDII style solution.
The population script (cron job written in Perl) will be expanded to handle the multiple instantiations of the regional GIISes.
The "stale information" problem.
The LDAP server uses a db that is stored in a set of flat files.
While we use one directory structure to serve the queries fill another directory structure with the new information obtained from the regional GIISes. After the new structure is filled we stop the ldap server and restart it using the new structure (which take about 1/4sec).
During the time the BDII is restarted no queries can be processed.
However, since we expect to run a BDII for a small number of RBs the frequency of queries is expected to be low (about 0.5 Hz max) and even if the RB is handling a missing BDII non gracefully this will have minimal impact on the overall reliability.
Time to start a BDII using the information already available is about 1/4 sec.
There are several ways to refine this approach, especially if a RB can be modified to use a list of BDIIs instead of a single one in which case the problem of a restarting BDII blocking the RB can be avoided.
Other measures that will improve the robustness of the information system:
To limit the amount of information published by the system we suggest to mimic the behavior that we will see after VOMS is introduced.
The information providers should stop publishing the DNs of all authorized users (this is a large fraction of the data).
The supported VOs are already published.
The user has to specify her VO membership in the jdl as a required runtime variable. During the matchmaking the RB will use this to decide which CEs are suitable for the job. The check against the DN of the user has to be removed. This is expected to be a minor change.
The timeouts in the MDS version provided by NorduGrid have to be tuned.

7. MDS Deployment

8. LCG-1 Distribution Packaging & Configuration Distribution
Service machines
LCFGng – either full or light version
Worker nodes – aim – allow sites to use existing tools
LCFGng
Installation scripts
Instructions
User interface
Pacman?
Distribution
Web site being set up now (updated from LCG-0)
Sets of rpm’s etc organised by service and machine type
Installation guides, release notes, etc., being written

9. Distribution Organisation
Contacts & Site Hierarchy:
A set of core sites (sites with sufficient human resources and dedication to actively participate in the deployment process) – Tier 1s
A set of secondary sites that deploy the software but rely on the core sites to lead – Tier 2s
In addition a partner relationship has to be defined between sites for mirroring of important services.
There are different criteria on grouping partner sites.
For services which are required to be accessible globally, partner sites should be network-wise separated as far as possible.
For other services, like file catalogs, some proximity is desirable.

10. Communication: List of core site representatives
The core sites have to maintain a list of contacts to the sysadmins of their secondary sites
Chain of responsibility has to be defined
Each site should provide a list of who is responsible for each service, and how problems are reported back
An IRC will be setup to provide rapid communication that can be logged
There might be other (better) tools around, but start with something that is well established and simple.

11. Roles: 1) Deployment Manager (DM) + Deputy (DDM):
Both located at one of the core sites.
The role is assigned for one deployment and will rotate between different individuals.
For major version changes (first deployment, non-backward upgrade) it is expected that they move to the same location for the duration of the deployment process.
If the role is rotated either the DM or the DDM must have acted in one of these roles before.
Rotation is important since it will spread the knowledge, helps to build a community and avoids overloading certain people.
The DM has to be the grid site manager of a core site (CDM)
2) Core site deployment manager(CDM)
Each is responsible for at least one site.
For the Tier 1s, they are responsible for steering the deployment of the lower levels of the grid.
3) Grid site manager (administrator): GSM
Represents the site during deployment and controls (or does) the actual installation/config work.
There is one GSM/site that is visible. There can of course be many working.
The DM/DDM are responsible that the agreed process is followed and have to escalate problems if they arise.
GSM/A’s report to CDMs and CDMs to the DM.
The DM's site for the first deployment is expected to start the process at least 3 days earlier.
As a result the top of the information system hierarchy will be already present, one RB, one CE(+WNs), SE, UI will be already available when the other sites follow.

12. Integration Issues LCG will try to be non-intrusive:
Will assume base OS is already installed
Provide installation & config tool for service nodes
Provide recipes for installation of WNs – assume sites will use existing tools to manage their clusters
No imposition of a particular batch system
As long as your batch system talks to Globus
(OK for LSF, PBS, Condor, BQS, FBSng)
No longer requirement for shared filesystem between gatekeeper and WNs
was a problem for AFS, NFS does not scale to large clusters
Information publishing
Define what information a site should provide (accounting, status, etc), rather than imposing tools
Auditing, accounting:
Developing basic tools to:
Correlate logs between gatekeeper, batch system, process log
Initially for audit trails, useful for basic resource use accounting
But … maybe some compromises in short term (2003)

13. Worker Node connectivity
In general (and eventually) it cannot be assumed that the cluster nodes will have connectivity to remote sites
Many clusters on non-routed networks (for many reasons)
Security issues
In any case this assumption will not scale
BUT…
To achieve this several things are necessary:
Some tools (e.g. replica management) must become services
Databases (e.g. conditions db) must either be replicated to each site (or equivalent), or proxy service, or …
Analysis models must take this into account
Again, short term exceptions (up to a point) possible
Current additions to LXbatch at CERN have this limitation