Download presentation
Presentation is loading. Please wait.
Published byMelinda Neal Modified over 9 years ago
1
QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh
2
Contents Background DataGrid: hardware status DataGrid: software status and future development Job Submission status and future development International Collaboration
3
Background Aim: To develop a distributed data storage and compute grid for quantum chromodynamics in the UK Currently a 4-site data grid –provides reliable distributed data storage, including a searchable metadata catalogue A job submission system is under development Key technologies used –Globus 2.0 –European Data Grid –eXist XML database
4
The Data Grid: Hardware The data storage grid has been up and running for several months –currently managing a few hundred gigabytes of data Consists of storage elements at 4 sites –Edinburgh, Liverpool, RAL and Swansea –may soon expand to Glasgow and Southampton QCDOC comes on-line 2004 –expect to store around 30 Tbytes of data –requested further space from GridPP –further disk space expected at Edinburgh
5
The Data Grid: Software Replication of files is currently managed by custom written software –built on Globus 2.0 –will consider replacing / integrating with existing software in the future Central control thread –ensures there are always at least two copies of each file, stored at different sites Replica catalogue –maps logical filenames to actual physical locations
6
The Metadata Catalogue Each file on the datagrid has an associated XML metadata document These documents are stored in the eXist open source XML database, where they can be searched using the XPath query language Java GUI provides a user friendly interface Metadata catalogue is itself stored on the datagrid
7
DataGrid: Future Development Review current available technologies for possible integration –after phase 1 of the job submission work Control Node –currently backed up on the data grid –develop second control node that allows read access to the datagrid automatically –allow full control node to be started manually on another system Integrate EDG software for virtual organisation management and security –already running on test bed system –currently being rolled out to main data grid system
8
Job Submission: Requirements Aim: To allow data generation/analysis jobs to be easily submitted to grid machines Integration with the existing datagrid is desirable Resource broking not essential –users normally know in advance on which machine a job should run –is however desirable – especially for analysis jobs Real-time job status monitoring important Must integrate with a diverse range of machines from normal (Linux) PCs to QCDOC User-friendly GUI or web portal a desirable
9
Job Submission: Technology As with the datagrid, requirements dictate a combination of existing software and purpose-built middleware –Globus toolkit to be used for low level access to grid resources and data –European Data Grid software to be used for virtual organisation management and security –Batch systems such as PBS will be integrated into the system –QCDgrid job submission software will build on these components, providing the interface and features that users need
10
Job Submission: Status Job submission system is being developed on a test grid independent from the main datagrid –Jobs can now be submitted to grid resources using a command line tool –Input files can be fetched automatically from the datagrid –Job output and input can be streamed to and from the user’s console allows jobs to be monitored allows interactive jobs to run on grid resources –All output files generated by the job are automatically brought back to the user’s local machine
11
Job Submission: Future Development First ‘users’ to test the software this month on test system Integrate with batch queuing systems such as PBS, loadleveler, Sun grid engine Improve the user interface Provide more advanced interaction with the datagrid –metadata operations –storing output automatically Deploy and test on “real” grid
12
International Collaboration: ILDG ILDG (International Lattice DataGrid) –A collaboration of scientists involved in lattice QCD from all over the world (UK, Japan, USA, France, Germany, Australia and other countries) –Working on standards to allow national datagrids to interoperate, for easier data sharing –Two working groups looking at different aspects of this goal: metadata and middleware
13
ILDG Working Groups ILDG metadata working group is developing a standard schema for describing QCD metadata –Likely to be based on UKQCD’s schema which is extensible and can be applied to various different data types ILDG middleware working group is developing a standard interface to grid data storage systems –Based on the SRM (Storage Resource Manager) specification –A basic implementation of SRM to interface with UKQCD’s datagrid has been written Together these standards will enable international data sharing
14
Summary Aim is to develop a grid for use by the UKQCD collaboration for storing data and performing computations A datagrid, including searchable metadata catalogue, is already in use A job submission system is in development Software based on Globus toolkit and European Data Grid middleware International efforts to develop standards for lattice QCD grids and data are ongoing
15
References QCDgrid Web Site –http://www.gridpp.ac.uk/qcdgrid ILDG Web Site –http://www.lqcd.org/ildg European Data Grid Project –http://www.eu-datagrid.org/ GridPP Web site –http://www.gridpp.ac.uk
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.