26 September 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000.

Slides:

Advertisements

Similar presentations

B A B AR and the GRID Roger Barlow for Fergus Wilson GridPP 13 5 th July 2005, Durham.

Advertisements

11th December 2002Tim Adye1 BaBar UK Grid Work Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting SLAC 11 th December 2002.

12th September 2002Tim Adye1 RAL Tier A Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Imperial College, London 12 th September 2002.

13th November 2002Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting University of Bristol 13 th November.

13 December 2000Tim Adye1 New KanGA Export Scheme Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Data Distribution Session 13 th December.

22nd January 2003Tim Adye1 Summary of Bookkeeping discussions at RAL Workshop Tim Adye Rutherford Appleton Laboratory Kanga Phone Meeting 22 nd January.

AppManager 7: Deep Technical Dive Tim Sedlack & Michi Schniebel Sr. Product Managers.

Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.

Batch Production and Monte Carlo + CDB work status Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.

Status of analysis & MC farms 30/08/2002 Concezio Bozzi INFN Ferrara.

Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage.

The B A B AR G RID demonstrator Tim Adye, Roger Barlow, Alessandra Forti, Andrew McNab, David Smith What is BaBar? The BaBar detector is a High Energy.

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.

11 Dec 2000F Harris Datagrid Testbed meeting at Milan 1 LHCb ‘use-case’ - distributed MC production

Exploiting the Grid to Simulate and Design the LHCb Experiment K Harrison 1, N Brook 2, G Patrick 3, E van Herwijnen 4, on behalf of the LHCb Grid Group.

DB Audit Expert v1.1 for Oracle Copyright © SoftTree Technologies, Inc. This presentation is for DB Audit Expert for Oracle version 1.1 which.

QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.

Channel Archiver Stats & Problems Kay Kasemir, Greg Lawson, Jeff Patton Presented by Xiaosong Geng (ORNL/SNS) March 2008.

25 February 2000Tim Adye1 Using an Object Oriented Database to Store BaBar's Terabytes Tim Adye Particle Physics Department Rutherford Appleton Laboratory.

BaBar WEB job submission with Globus authentication and AFS access T. Adye, R. Barlow, A. Forti, A. McNab, S. Salih, D. H. Smith on behalf of the BaBar.

IRODS performance test and SRB system at KEK Yoshimi KEK Building data grids with iRODS 27 May 2008.

Alexandre A. P. Suaide VI DOSAR workshop, São Paulo, 2005 STAR grid activities and São Paulo experience.

GLAST LAT ProjectDOE/NASA Baseline-Preliminary Design Review, January 8, 2002 K.Young 1 LAT Data Processing Facility Automatically process Level 0 data.

LHCb Applications and GRID Integration Domenico Galli Catania, April 9, st INFN-GRID Workshop.

2nd April 2001Tim Adye1 Bulk Data Transfer Tools Tim Adye BaBar / Rutherford Appleton Laboratory UK HEP System Managers’ Meeting 2 nd April 2001.

Testing Session Testing Team-Release Management Team.

©2010 John Wiley and Sons Chapter 12 Research Methods in Human-Computer Interaction Chapter 12- Automated Data Collection.

QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Jean-Yves Nief CC-IN2P3, Lyon HEPiX-HEPNT, Fermilab October 22nd – 25th, 2002.

Cosener’s House – 30 th Jan’031 LHCb Progress & Plans Nick Brook University of Bristol News & User Plans Technical Progress Review of deliverables.

8th November 2002Tim Adye1 BaBar Grid Tim Adye Particle Physics Department Rutherford Appleton Laboratory PP Grid Team Coseners House 8 th November 2002.

Nick Brook Current status Future Collaboration Plans Future UK plans.

March 6, 2009Tofigh Azemoon1 Real-time Data Access Monitoring in Distributed, Multi Petabyte Systems Tofigh Azemoon Jacek Becla Andrew Hanushevsky Massimiliano.

BaBar Data Distribution using the Storage Resource Broker Adil Hasan, Wilko Kroeger (SLAC Computing Services), Dominique Boutigny (LAPP), Cristina Bulfon.

4/20/02APS April Meeting1 Database Replication at Remote sites in PHENIX Indrani D. Ojha Vanderbilt University (for PHENIX Collaboration)

Data Distribution and Management Tim Adye Rutherford Appleton Laboratory BaBar Computing Review 9 th June 2003.

Status of the LHCb MC production system Andrei Tsaregorodtsev, CPPM, Marseille DataGRID France workshop, Marseille, 24 September 2002.

PHENIX and the data grid >400 collaborators Active on 3 continents + Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.

19th September 2003Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting Royal Holloway 19 th September 2003.

25th October 2006Tim Adye1 RAL Tier A Tim Adye Rutherford Appleton Laboratory BaBar UK Physics Meeting Queen Mary, University of London 25 th October 2006.

SkimData and Replica Catalogue Alessandra Forti BaBar Collaboration Meeting November 13 th 2002 skimData based replica catalogue RLS (Replica Location.

October 8, 2002P. Nilsson, SPD General Meeting1 Paul Nilsson, SPD General Meeting, Oct. 8, 2002 New tools and software updates Test beam analysis Software.

GridPP11 Liverpool Sept04 SAMGrid GridPP11 Liverpool Sept 2004 Gavin Davies Imperial College London.

GridPP Collaboration Meeting 5 th November 2001 Dan Tovey, University of Sheffield Non-LHC and Non-US-Collider Experiments’ Requirements Dan Tovey, University.

A B A B AR InterGrid Testbed Proposal for discussion Robin Middleton/Roger Barlow Rome: October 2001.

Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center

PHENIX and the data grid >400 collaborators 3 continents + Israel +Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.

CHEP 2000, Padova, Feb 2000 Operational Experience with the B A B AR Database David R. Quarrie Lawrence Berkeley National Laboratory for B A B AR Computing.

On the Verge of One Petabyte – the Story Behind the BaBar Database System Jacek Becla Stanford Linear Accelerator Center For the BaBar Computing Group.

11th November 2002Tim Adye1 Distributed Analysis in the BaBar Experiment Tim Adye Particle Physics Department Rutherford Appleton Laboratory University.

| nectar.org.au NECTAR TRAINING Module 9 Backing up & Packing up.

11th April 2003Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting Liverpool 11 th April 2003.

BaBar and the GRID Tim Adye CLRC PP GRID Team Meeting 3rd May 2000.

15 December 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 15 th December 2000.

Distributing Data around the BaBar collaboration’s Objectivity Federations Emanuele Leonardi - I.N.F.N. Roma Simon Patton - L.B.N.L. on behalf of the BaBar.

Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,

D0 File Replication PPDG SLAC File replication workshop 9/20/00 Vicky White.

CDF SAM Deployment Status Doug Benjamin Duke University (for the CDF Data Handling Group)

Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.

11th September 2002Tim Adye1 BaBar Experience Tim Adye Rutherford Appleton Laboratory PPNCG Meeting Brighton 11 th September 2002.

Existing Perl/Oracle Pipeline

Artem Trunov and EKP team EPK – Uni Karlsruhe

Chapter 12: Automated data collection methods

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System Zaihua Ji Doug Schuster Steven Worley Computational.

Using an Object Oriented Database to Store BaBar's Terabytes

Gridifying the LHCb Monte Carlo production system

Kanga Tim Adye Rutherford Appleton Laboratory Computing Plenary

Presentation transcript:

26 September 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000

26 September 2000Tim Adye2 KanGA Exports Current exports Developments Obectivity Data Exports Objectivity SP Imports Objectivity Data Distribution Tools Plans Parallel Session on Thursday morning...

26 September 2000Tim Adye3 KanGA Exports Currently 1.8 TB on disk at SLAC 8 sites take everything, 6 a subset [CCG Survey] European sites copy from each other to reduce transatlantic traffic Sites can elect to just take the skims of local interest Data copied using rsync/syncslac mirroring tools Easy to set up and use Various improvements in tools But does not scale so well Takes ~3 hours just to scan disk for updates (130k files) Difficult to specify subset of data to copy (uses file path)

26 September 2000Tim Adye4 KanGA Tools Want to control export using using a database Use existing collection catalogue (skimData) Catalogue mirrored at remote sites Don’t have to log onto SLAC to use skimData Generate TCL files just for files already local Outside SLAC, Use mySQL, so don’t have to buy/install Oracle Which skims/runs/processings to copy selected using skimData

26 September 2000Tim Adye5 KanGA Export Tools Status skimData catalogue mirrored SLAC -> RAL -> UK Universities (under test) [Alessandra Forti] Export control being commissioned [AF/TA] A similar approach is in use at CASPUR Includes local backup [Alvise Dorigo, Emanuele Leonardi] Investigate alternative file copy tools [TA et al] rsync vs. multi-threaded tools: bbftp (IN2P3), sfcp (SLAC), gsiftp (GLOBUS)

26 September 2000Tim Adye6 Objectivity Export All micro data exported to CC-IN2P3 [A-M Lutz] Rome and RAL now take just KanGA All exports go via the network bbftp to optimise bulk data transport [Gilles Farrache] Various optimisations implemented Data/conditions export now integrated into OPR sweeps [Adil Hasan, Artem Trunov] Skim exports still come from analboot2 (more tricky) Fewer local file copies on export (SLAC) and import (CC-IN2P3) [J-N Albert, Dominique Boutigny]

26 September 2000Tim Adye7 SP Imports to SLAC MC Imports being automated Production Centres responsible for export and copying to SLAC Automatic procedure being developed [Daniele Andreotti, Emanuele Leonardi] Automatic procedure for SLAC import [Cristina Bulfon et al] Starts when files arrive on disk Check files, import to SP3 federation, update bfreport See Jim Smith’s talk…

26 September 2000Tim Adye8 Objectivity Data Distribution Tools (Recent improvements -- only a very brief summary) New “collection metadata” information [Moreno Marzolla] Stores collection DB file correspondences Bypass slow scan to find which files to export Error checking and reporting improvements [TA] Objectivity Catalogue browser [Yemi Adesanya] Java GUI and “batch mode” selection of DB files to export

26 September 2000Tim Adye9 Objectivity Data Distribution Tools (Current work -- only a very brief summary) Single event / run / skim export [MM] Creates and exports “All-In-One” database files, with just the requested data Can be used for raw/reco/micro export of small samples of events Redesign of Data Distribution Tools in Java [JNA] Uses existing (C++) Objectivity access code Optimised for specific requirements Eg. Bulk import over the net (CC-IN2P3)

26 September 2000Tim Adye10 Plans Wait for Computing Model report (tomorrow ) Whichever way we go, we need to continue to support Objectivity exports (SP, access to samples of reco) KanGA exports (at least until Bdb/Objectivity meets requirements) Automatic bulk exports via the net Exports of small samples for individual users

26 September 2000Tim Adye11 Parallel Session Data Distribution / Simulation Production meeting on Thursday morning 8:30, Orange Room See HN for agenda and phone-in details datadist / simuprod HN groups