1. 26 September 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000

2. 26 September 2000Tim Adye2 KanGA Exports Current exports Developments Obectivity Data Exports Objectivity SP Imports Objectivity Data Distribution Tools Plans Parallel Session on Thursday morning...

3. 26 September 2000Tim Adye3 KanGA Exports Currently 1.8 TB on disk at SLAC 8 sites take everything, 6 a subset [CCG Survey] European sites copy from each other to reduce transatlantic traffic Sites can elect to just take the skims of local interest Data copied using rsync/syncslac mirroring tools Easy to set up and use Various improvements in tools But does not scale so well Takes ~3 hours just to scan disk for updates (130k files) Difficult to specify subset of data to copy (uses file path)

4. 26 September 2000Tim Adye4 KanGA Tools Want to control export using using a database Use existing collection catalogue (skimData) Catalogue mirrored at remote sites Don’t have to log onto SLAC to use skimData Generate TCL files just for files already local Outside SLAC, Use mySQL, so don’t have to buy/install Oracle Which skims/runs/processings to copy selected using skimData

5. 26 September 2000Tim Adye5 KanGA Export Tools Status skimData catalogue mirrored SLAC -> RAL -> UK Universities (under test) [Alessandra Forti] Export control being commissioned [AF/TA] A similar approach is in use at CASPUR Includes local backup [Alvise Dorigo, Emanuele Leonardi] Investigate alternative file copy tools [TA et al] rsync vs. multi-threaded tools: bbftp (IN2P3), sfcp (SLAC), gsiftp (GLOBUS)

6. 26 September 2000Tim Adye6 Objectivity Export All micro data exported to CC-IN2P3 [A-M Lutz] Rome and RAL now take just KanGA All exports go via the network bbftp to optimise bulk data transport [Gilles Farrache] Various optimisations implemented Data/conditions export now integrated into OPR sweeps [Adil Hasan, Artem Trunov] Skim exports still come from analboot2 (more tricky) Fewer local file copies on export (SLAC) and import (CC-IN2P3) [J-N Albert, Dominique Boutigny]

7. 26 September 2000Tim Adye7 SP Imports to SLAC MC Imports being automated Production Centres responsible for export and copying to SLAC Automatic procedure being developed [Daniele Andreotti, Emanuele Leonardi] Automatic procedure for SLAC import [Cristina Bulfon et al] Starts when files arrive on disk Check files, import to SP3 federation, update bfreport See Jim Smith’s talk…

8. 26 September 2000Tim Adye8 Objectivity Data Distribution Tools (Recent improvements -- only a very brief summary) New “collection metadata” information [Moreno Marzolla] Stores collection DB file correspondences Bypass slow scan to find which files to export Error checking and reporting improvements [TA] Objectivity Catalogue browser [Yemi Adesanya] Java GUI and “batch mode” selection of DB files to export

9. 26 September 2000Tim Adye9 Objectivity Data Distribution Tools (Current work -- only a very brief summary) Single event / run / skim export [MM] Creates and exports “All-In-One” database files, with just the requested data Can be used for raw/reco/micro export of small samples of events Redesign of Data Distribution Tools in Java [JNA] Uses existing (C++) Objectivity access code Optimised for specific requirements Eg. Bulk import over the net (CC-IN2P3)

10. 26 September 2000Tim Adye10 Plans Wait for Computing Model report (tomorrow ) Whichever way we go, we need to continue to support Objectivity exports (SP, access to samples of reco) KanGA exports (at least until Bdb/Objectivity meets requirements) Automatic bulk exports via the net Exports of small samples for individual users

11. 26 September 2000Tim Adye11 Parallel Session Data Distribution / Simulation Production meeting on Thursday morning 8:30, Orange Room See HN for agenda and phone-in details datadist / simuprod HN groups