11/01/20081 Data simulator status CCRC’08 Preparatory Meeting Radu Stoica, CERN* 11 th January 2007 * On leave from IFIN-HH

11/01/20082 How the data challenge works (Online view) A set of MDF files are produced in advance. They are copied to the Online storage in the pit. A data simulator will create symbolic links to files and register them in the Run Database. Files are copied to CASTOR and added to the LCG File Catalog. The files are added to the LHCb Bookkeeping database. Files are available for offline reconstruction in GRID. After migration, files (or rather the links to them) can be deleted in the Online storage

11/01/20083 Two possibilities for creating a file generator from the MDF files: 1. Register them directly as if created by the data writer a) Preparatory step (necessary to avoid duplication of IO): Data Simulator Online SAN MDF Files Script Database Calculates the necessary checksums Saves other file information copy read

11/01/20084 Two possibilities for creating a file generator from the MDF files: 1. Register them directly as if created by the data writer b) File generator: Data Simulator Online SAN Data Simulator Database 4. Creates runs 5. Generates file entries (name, directory structure, etc) 6. Closes runs 2. Directory structure 3. Symlinks Online Run DB 1. MDF file data Replaces ECS

11/01/20085 Two possibilities for creating a file generator from the MDF files: 1. Register them directly as if created by the data writer Pros/Cons: Easy to implement/use. No need to multiply IO to Online SAN Less software components involved (no special control system) Small number of machines required (1-2 servers) Easy to control rate (just add a new file entry at desired interval) Questions: Is the fact that the event time in the MDF files will be always the same a problem? (e.g. because of the conditions db) Data Simulator

11/01/20086 Two possibilities for creating a file generator from the MDF files: 2. Replace event builder and sent them directly to the streaming and formatting layer: Data Simulator Online SAN Data Simulator Extract events from MDF files Online Run DB replaces farm Streaming Formatting Data Writer ECS ? ?

11/01/20087 Two possibilities for creating a file generator from the MDF files: 2. Replace the farm and sent them directly to the streaming and formatting layer Pros/Cons/Observations: Requires 3x IO to storage (reading, writing, reading). At least 6 servers needed (if we want to scale up the system). More software components tested (also the ECS, Streaming/Formatting layer, Data Writer are involved). The rate can also be controlled by monitoring the writing time of a file. Data Simulator

11/01/20088 A new LUN was created on the Online SAN (30 TBs), the same size as the CASTOR RAW pool. The IO file migration requirements are: –Minimum 75 MB/s. This is the expected average rate during the data challenge. –Maximum 150 MB/s as agreed with IT. The actual IO requirements are multiplied depending on how many times a file is to be read/written. Using the 10 Gb/s network connection to CASTOR. The store03, store04 servers will be used to migrate files. Storage

11/01/20089 There is a need to create a clear picture of the whole process and decide responsibilities. Can the results of today’s discussion be written to some document? We should have a test setup as soon as possible. What is needed exactly for this? Suggestions/Questions