Setting up a Pan-European Datagrid using QCDgrid technology Chris Johnson, James Perry, Lorna Smith and Jean-Christophe Desplat EPCC, The University Of Edinburgh The ENACTS “Demonstrator”

This Talk A summary of the title: –ENACTS Demonstrator –Pan-European Datagrid –QCDgrid.

ENACTS European Network for Advanced Computing Technology for Science. EC-funded project with 14 members. Started in Attempt to ensure that Europe did not lag behind US in grid technology. ENACTS originally consisted of many reports reports with little technical work.

14 ENACTS partners Please see:

ENACTS Demonstrator: Partners involved EPCC, Edinburgh, UK –Chris Johnson –Jean-Christophe Desplat –James Perry Parallab, Bergen, Norway –Jacko Koster –Jan-Frode Myklebust –Csaba Anderlik TCD, Dublin, Ireland –Geoff Bradley –Bob Crosbie.

ENACTS Demonstrator… Objective –“To enable the formation of a pan-European HPC metacentre…”. The Demonstrator is part of Phase II of the activity and its specific objective is –“to draw together the results from all of the Phase I technology studies and evaluate their practical consequences for operating a pan- European metacentre and constructing a best- practice model for collaborative working amongst individual facilities”.

ENACTS Demonstrator ENACTS Phase I –consisted mainly of reports ENACTS Phase II –contained the Demonstrator activity Phase I identified technologies such as –Globus, replica management, LDAP database and XML metadata. All these technologies are inherent in the QCDgrid system.

metacentre A “virtual organisation” with data described by metadata. –Users submit data from any site –The data is stored on “the grid” –All data is stored reliably –All data is easy to retrieve.

Our Demonstrator Set-up QCDgrid across the 3-sites to create our metacentre. Use a genuine scientific scenario. Use an XML schema for meta-data. Ensure the data is portable between the systems involved.

Summary so far… ENACTS demonstrator project is an EC funded project involving 3 partners attempting to set up a pan- European metadata centre using QCDgrid technology.

What is QCDgrid? It’s not QCD-specific!! QCDgrid was written to manage the QCD data belonging to the UK QCD community (UKQCD) –2 previous All-Hands talks (James Perry,EPCC). The original grid consisted of 6 geographically dispersed sites. Around 5 terabytes of data. The amount of data is expected to grow dramatically when QCDOC comes online later in 2004.

What is QCDgrid? QCDgrid is a layer of software written on top of the Globus Toolkit. –Uses security infrastructure and basic grid operations such as data transfer –also uses more advanced features such as the replica catalogue.

How does QCDgrid work? A control thread runs on one storage element –constantly scans grid –ensures all storage elements are working –ensures all files are stored in at least 2 suitable locations. When a new file is added it is rapidly replicated across grid onto 2 or more geographically separate sites.

QCDgrid:Dealing with node loss If a storage element is lost unexpectedly, all files that were held on the failed system are replicated elsewhere. QCDgrid can cope with loss of entire site. If the control node is lost – control reverts to a secondary node.

How is QCDgrid used? Assuming QCDgrid is set up and the control thread and metadata database are running… and that each user has a valid certificate… User submits the usual initialisation commands –grid-proxy-init –source the correct set up files –sets a few paths, classpaths, etc.

Submitting files Command line User submits a file (datafile.dat) –put-file-on-qcdgrid datafile.dat AND an accompanying metadata file which describes the above data file –put-file-on-qcdgrid datafile.xml –exist:/db>put datafile.xml

Submitting files (better still…) Using the GUI User runs the Java GUI and submits both data file and metadata file at the same time. metadata file has a tag for the name of the corresponding data file –marries the two –every data file should have an associated metadata file.

Metadata Browser Can submit, search and retrieve data using this Java browser.

Metadata/Datagrid Integration QCDgrid software deals with storage and replication of data. eXist database deals with cataloging of data using metadata. All can be controlled using command line or GUI.

More on QCDgrid Other commands available with QCDgrid –qcdgrid-list lists all files on grid. –get-file-from-qcdgrid retrieves files from grid. –i-like-this-file attempts to store a file local to the user. There are also several commands for administering nodes, etc.

QCDgrid Sites ? QCDgrid ENACTS ?

Our depoyment of QCDgrid UK (all using e-science CA) -> Europe (all using different CAs). Moving from a homogenous Linux environment to a mixed one (Linux/Solaris). Moving from Globus Toolkit (GT) > GT2.4.

How difficult was it? Certificates –Some certificate issuers took several weeks to issue certificates. –Different policies on issuing certificates, e.g. non-human users (project accounts). –Not too many difficulties using multiple certificates.

How difficult was it? Moving to a heterogeneous environment. –Installing of Globus 2.x is difficult on Solaris – led to the Solaris node being unable to submit data. A few minor problems getting system specific functions to work (e.g. df command). Usual minor compilation issues – did require gcc compiler.

How difficult was it? Globus –This presented the biggest difficulty! –Installation difficulties and firewall issues several months before a “helloworld” job would run from any site to any other. –Migrating from GT 2.0 -> GT 2.4 Major difficulties! Had to re-write the replica schema. Remove some error-handling functionality.

Users – Scientific scenario Appealed to QCD –Given more time we could have found a different discipline. Two users from TCD as well as those involved in the project itself. Code used was MILC code. Monte-Carlo simulation to investigate string-breaking. Three Monte-Carlo chains, one on each node.

User feedback Generally impressed with functionality. Some frustration in getting certificates. Difficulty persuading users to use metadata, although agreed it was useful. Would make more use of file-sharing using such a system. Liked machine-independent data. Wanted grid to do job submission.

Conclusions We have created a pan-European datagrid (metacentre) using QCDgrid technology. The systems works well… …Globus is the limiting factor. Users were impressed with the system in use. The system was not tested with many users but we can see no reason why it would not scale to many users/nodes if Globus allows.

Acknowledgements James Perry. Jean-Christophe Desplat, Jacko Koster, Jan-Frode Myklebust, Csaba Anderlik, Geoff Bradley, Bob Crosbie. Craig McNeile and Bálint Joó. Mike Peardon and Jimmy Juge.

References ENACTS – QCDgrid – –code: MILC code –