1. Status and plans for bookkeeping system and production tools
Eric van Herwijnen
Thursday, 19 september 2002

2. Contents Status remote centres & datachallenge plans
DataGrid status and plans
Data management and production tools
Job submission
Bookkeeping
Configuration
Data production and monitoring
Roadmap and conclusions

3. Status remote centres Center No. of CPU’s (1 GHz) Production tools
CERN
~ 400
new
Lyon
60 +
Liverpool
~ 120
Imperial College
~ 100
DataGrid
~ 20
RAL
~ 300
old
Bologna
~ 200
Nikhef
Bristol
Edinburgh
Cambridge
~ 15
Oxford
~ 10
Moscow
~ 40
Rio
Total
~ 1000 (outside CERN)

4. Physics Data Challenge plans
MC production = Physics Data Challenge
Available capacity seems to match requirements
Planning:
Preproduction: mid Dec 2002 – mid Jan 2003
Production: Feb – May 2003

5. DataGrid status and plans
Installation operational
Long job problem fixed
Long file transfer problem (~ 1 Gb)
New production tools being installed
Test:
Run 500 event MC generation
Store on SE
Recover logs and histograms to CERN
Run reconstruction. Output to SE. Recover log files and histos.
Write recon output to mass store (Castor)
Read Castor data with an analysis job outside Grid

6. Data Management

7. Job submission New tools designed and written by A. Tsaregorodtsev
Site dependencies concentrated in 3 scripts
Simple installation procedure:
AFS independent
Standard directory structure
No (or very little) java
More flexible updating of bookkeeping database
Remote centers should migrate to new system

9. Job submission, future Data Production Job.opts DB Bookkeeping DB
Production done
Modify
Create job(s)
script
Prod.Mgr
Configuration DB
Build new configuration
Selection of Defaults
Information Flow

10. Job submission, future
Workflow to be implemented (template scripts for each step, parameters added via web page)
Prototype exists (M. Frank)

11. Bookkeeping

12. Bookkeeping Most components ready (S. Ponce, F. Loverre)
Structure of database independent of tools, easy to add new datatypes, identify productions, replicas of datasets
Database (Oracle) filled by a Java server via XML files
API implemented in Java and Python
Migration to new database transparent for user
Final tests under way before production

13. Configuration Prototype of configuration database exists
Need to integrate with job submission tools (5 line python script?)
Prototype of GUI to view the database developed by G. Klamke (summer student)
Need to create tool to add configurations to database, and integrate with Ganga

15. Data Production and Monitoring

16. Data production and monitoring
Current PVSS system needs to be brought beyond the prototype stage
Cosmetic changes, cleanup of dead entries, speed
Adapt to new job submission tools
Alarms should really be alarms, add corrective action
Migrate to new version of PVSS
Web interface
Production database + interface need to be created
Data quality check tools need to be reviewed and created
Ongoing activity

17. Roadmap and conclusions
Installation of new job submission tools: oct-nov 2002
New bookkeeping db: dec 2002
Integration of job configuration db with job submission tools: feb 2003
Integration of job configuration db with Ganga: summer 2003
Creation of data production db, integration with monitoring system: summer-fall 2003