15 December 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 15 th December 2000.

Slides:

Advertisements

Similar presentations

11th December 2002Tim Adye1 BaBar UK Grid Work Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting SLAC 11 th December 2002.

Advertisements

12th September 2002Tim Adye1 RAL Tier A Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Imperial College, London 12 th September 2002.

13th November 2002Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting University of Bristol 13 th November.

13 December 2000Tim Adye1 New KanGA Export Scheme Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Data Distribution Session 13 th December.

22nd January 2003Tim Adye1 Summary of Bookkeeping discussions at RAL Workshop Tim Adye Rutherford Appleton Laboratory Kanga Phone Meeting 22 nd January.

Batch Production and Monte Carlo + CDB work status Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.

GRID DATA MANAGEMENT PILOT (GDMP) Asad Samar (Caltech) ACAT 2000, Fermilab October , 2000.

Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage.

Background Info The UK Mirror Service provides mirror copies of data and programs from many sources all over the world. This enables users in the UK to.

Oxford Jan 2005 RAL Computing 1 RAL Computing Implementing the computing model: SAM and the Grid Nick West.

The B A B AR G RID demonstrator Tim Adye, Roger Barlow, Alessandra Forti, Andrew McNab, David Smith What is BaBar? The BaBar detector is a High Energy.

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System 1 Zaihua Ji Doug Schuster Steven Worley Computational.

Installing and Maintaining ISA Server. Planning an ISA Server Deployment Understand the current network infrastructure Review company security policies.

Agenda  Overview  Configuring the database for basic Backup and Recovery  Backing up your database  Restore and Recovery Operations  Managing your.

Slide 1 of 9 Presenting 24x7 Scheduler The art of computer automation Press PageDown key or click to advance.

SubVersioN – the new Central Service at DESY by Marian Gawron.

October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.

Week 2CSC 1511 Introduction to Information Systems CSC 151 Week 2.

AliEn uses bbFTP for the file transfers. Every FTD runs a server, and all the others FTD can connect and authenticate to it using certificates. bbFTP implements.

Chapter 18: Windows Server 2008 R2 and Active Directory Backup and Maintenance BAI617.

Module 13: Maintaining Software by Using Windows Server Update Services.

3 Sept 2001F HARRIS CHEP, Beijing 1 Moving the LHCb Monte Carlo production system to the GRID D.Galli,U.Marconi,V.Vagnoni INFN Bologna N Brook Bristol.

2nd April 2001Tim Adye1 Bulk Data Transfer Tools Tim Adye BaBar / Rutherford Appleton Laboratory UK HEP System Managers’ Meeting 2 nd April 2001.

03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.

Computing Infrastructure Status. LHCb Computing Status LHCb LHCC mini-review, February The LHCb Computing Model: a reminder m Simulation is using.

Jean-Yves Nief CC-IN2P3, Lyon HEPiX-HEPNT, Fermilab October 22nd – 25th, 2002.

TELE 301 Lecture 10: Scheduled … 1 Overview Last Lecture –Post installation This Lecture –Scheduled tasks and log management Next Lecture –DNS –Readings:

1 Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory.

8th November 2002Tim Adye1 BaBar Grid Tim Adye Particle Physics Department Rutherford Appleton Laboratory PP Grid Team Coseners House 8 th November 2002.

Transferring VMS SAS/SPSS Data to UNIX Jeff Hayes Social Science Computing Co-op February 26, 1999.

IT 456 Seminar 5 Dr Jeffrey A Robinson. Overview of Course Week 1 – Introduction Week 2 – Installation of SQL and management Tools Week 3 - Creating and.

1 LCH.Clearnet – Sysgem AG Automated Production Manager Requests Discussion, 24 Aug 2011 WebEx Conference.

Data Distribution and Management Tim Adye Rutherford Appleton Laboratory BaBar Computing Review 9 th June 2003.

MAGDA Roger Jones UCL 16 th December RWL Jones, Lancaster University MAGDA  Main authors: Wensheng Deng, Torre Wenaus Wensheng DengTorre WenausWensheng.

SAM Installation Lauri Loebel Carpenter and the SAM Team February

PHENIX and the data grid >400 collaborators Active on 3 continents + Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.

19th September 2003Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting Royal Holloway 19 th September 2003.

25th October 2006Tim Adye1 RAL Tier A Tim Adye Rutherford Appleton Laboratory BaBar UK Physics Meeting Queen Mary, University of London 25 th October 2006.

SkimData and Replica Catalogue Alessandra Forti BaBar Collaboration Meeting November 13 th 2002 skimData based replica catalogue RLS (Replica Location.

GridPP Collaboration Meeting 5 th November 2001 Dan Tovey, University of Sheffield Non-LHC and Non-US-Collider Experiments’ Requirements Dan Tovey, University.

A B A B AR InterGrid Testbed Proposal for discussion Robin Middleton/Roger Barlow Rome: October 2001.

Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center

26 September 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000.

Jens G Jensen RAL, EDG WP5 Storage Element Overview DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003.

PHENIX and the data grid >400 collaborators 3 continents + Israel +Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

11th November 2002Tim Adye1 Distributed Analysis in the BaBar Experiment Tim Adye Particle Physics Department Rutherford Appleton Laboratory University.

UTA MC Production Farm & Grid Computing Activities Jae Yu UT Arlington DØRACE Workshop Feb. 12, 2002 UTA DØMC Farm MCFARM Job control and packaging software.

11th April 2003Tim Adye1 RAL Tier A Status Tim Adye Rutherford Appleton Laboratory BaBar UK Collaboration Meeting Liverpool 11 th April 2003.

BaBar and the GRID Tim Adye CLRC PP GRID Team Meeting 3rd May 2000.

Distributed Logging Facility Castor External Operation Workshop, CERN, November 14th 2006 Dennis Waldron CERN / IT.

SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.

10 May 2001WP6 Testbed Meeting1 WP5 - Mass Storage Management Jean-Philippe Baud PDP/IT/CERN.

Distributing Data around the BaBar collaboration’s Objectivity Federations Emanuele Leonardi - I.N.F.N. Roma Simon Patton - L.B.N.L. on behalf of the BaBar.

Feedback from CMS Andrew Lahiff STFC Rutherford Appleton Laboratory Contributions from Christoph Wissing, Bockjoo Kim, Alessandro Degano CernVM Users Workshop.

CMS: T1 Disk/Tape separation Nicolò Magini, CERN IT/SDC Oliver Gutsche, FNAL November 11 th 2013.

Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.

Acronyms GAS - Grid Acronym Soup, LCG - LHC Computing Project EGEE - Enabling Grids for E-sciencE.

CDF SAM Deployment Status Doug Benjamin Duke University (for the CDF Data Handling Group)

Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.

11th September 2002Tim Adye1 BaBar Experience Tim Adye Rutherford Appleton Laboratory PPNCG Meeting Brighton 11 th September 2002.

BaBar & Grid Eleonora Luppi for the BaBarGrid Group TB GRID Bologna 15 febbraio 2005.

Backups for Azure SQL Databases and SQL Server instances running on Azure Virtual Machines Session on backup to Azure feature (manual and managed) in SQL.

Printer Admin Print Job Manager

A Web-Based Data Grid Chip Watson, Ian Bird, Jie Chen,

Operational Dataset Update Functionality Included in the NCAR Research Data Archive Management System Zaihua Ji Doug Schuster Steven Worley Computational.

Bridging the Gap Between Technology and Business

Web100 at SLAC Doug Chang, Warren Matthews, Les Cottrell (SLAC).

Kanga Tim Adye Rutherford Appleton Laboratory Computing Plenary

Presentation transcript:

15 December 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 15 th December 2000

15 December 2000Tim Adye2 SP Exports SP Imports Objectivity Data Exports Kanga Data Exports Lots of other excellent and vital work going on behind the scenes…

15 December 2000Tim Adye3 SP Exports SP data generated at remote sites has to be exported from site Production Federation, copied to SLAC, and then imported to SLAC Federation Old system manual, tedious, and error-prone Production stopped Arcane BdbDistTools commands executed Long wait for these to complete before restarting production Files copied to SLAC

15 December 2000Tim Adye4 Automatic SP Export New tool by Emanuele Leonardi & Daniele Andreotti Prototype tested at a few sites Automated export (standard BdbDistTools) and ftp GUI control New version performs export in parallel with production Only “closed” (full) databases exported Needs less staging space 100+ GB -> 20 GB Transfer to SLAC now uses bbftp. Will be tested on next Production cycle at Rome

15 December 2000Tim Adye5 SP Imports Production sites copy their data to datamove3 for import into SLAC Federation Import procedure now automated [Cristina Bulfon] Runs from a cron job Checks for new export from Production sites Maintains e-logbook of operations [Lawrence Mount] Requires production sites to follow simple protocol See DataDist HN 111 for details Still need to improve error handling

15 December 2000Tim Adye6 Objectivity Data Exports Automated bulk exports to IN2P3 continue Further efficiency improvements introduced BdbServer, bulk export tool can also simplify small user exports: Bypasses many common problems Finding suitable export space (and tidying it up afterwards!), authorisation problems, complex options Simple interface Send list of DB IDs –(eg. from colldb list; metadata list in future) Data placed on ftp-accessible disk See Data Distribution Web page for details

15 December 2000Tim Adye7 Kanga exports 228k Kanga files, using 4 TB – and still growing! Current procedure (syncslac / rsync) is too slow  >4 hours, just scanning directories for new files Data transfer not optimised for WAN New procedure [Alessandra Forti, TJA] Uses skimData catalogue to find new files (~10s) Uses optimised “ftp” tools (bbftp / sfcp) Larger TCP/IP window size Multiple streams for each file Eg. bbftp SLAC RAL gives x10 improvement! Your mileage may vary – a lot

15 December 2000Tim Adye8 Kanga export tools - Status Tested SLAC -> RAL: 2 TB -> 3.1 TB A couple more sites recruited as “guinea pigs” More welcome! New tools for local management under development [Alvise Dorigo] Backup/archive to tape Delete old files Controlled by changes in skimData database

15 December 2000Tim Adye9 Conclusion Automation and efficiency improvements SP Exports – under test SP Imports – in production Simplified small user Objectivity exports Available for use Much faster Kanga export procedure Being deployed