Status of OPAL data Matthias Schröder

Slides:



Advertisements
Similar presentations
31/03/00 CMS(UK)Glenn Patrick What is the CMS(UK) Data Model? Assume that CMS software is available at every UK institute connected by some infrastructure.
Advertisements

Michigan State University 4/15/ Simulation for Level 2 James T. Linnemann Michigan State University NIU Triggering Workshop October 17, 1997.
© Vera Castleman Software Grade 10. What is software? A program is a collection of instructions to do a job. Programs are collectively known as SOFTWARE.
11 BACKING UP AND RESTORING DATA Chapter 4. Chapter 4: BACKING UP AND RESTORING DATA2 CHAPTER OVERVIEW Describe the various types of hardware used to.
Objectivity Data Migration Marcin Nowak, CERN Database Group, CHEP 2003 March , La Jolla, California.
ATLAS Analysis Model. Introduction On Feb 11, 2008 the Analysis Model Forum published a report (D. Costanzo, I. Hinchliffe, S. Menke, ATL- GEN-INT )
DATA PRESERVATION IN ALICE FEDERICO CARMINATI. MOTIVATION ALICE is a 150 M CHF investment by a large scientific community The ALICE data is unique and.
Data Preservation Best Practices for preserving your research data for future reuse The goal of data preservation is to ensure that your data is in a sustainable.
CERN/IT/DB Multi-PB Distributed Databases Jamie Shiers IT Division, DB Group, CERN, Geneva, Switzerland February 2001.
Biostatistics Analysis Center Center for Clinical Epidemiology and Biostatistics University of Pennsylvania School of Medicine Minimum Documentation Requirements.
L3 Filtering: status and plans D  Computing Review Meeting: 9 th May 2002 Terry Wyatt, on behalf of the L3 Algorithms group. For more details of current.
Status of Hall C 6 GeV Analysis Software Robust Fortran/CERNLIB code, “ENGINE”, for analysis of HMS/SOS coincidence and single arm experiments that has.
BES III core/reconstruction software planning for Release 3.0 BES III core/reconstruction software planning for Release 3.0 Li Weidong
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
November 7, 2001Dutch Datagrid SARA 1 DØ Monte Carlo Challenge A HEP Application.
Building a distributed software environment for CDF within the ESLEA framework V. Bartsch, M. Lancaster University College London.
Data Import Data Export Mass Storage & Disk Servers Database Servers Tapes Network from CERN Network from Tier 2 and simulation centers Physics Software.
As of 28 Juni 2005Getting Starged with GEM - Shuei Yamada 1 Getting Started with GEM Shuei YAMADA ICEPP, University of Tokyo What is GEM? Before you start.
File Systems (1). Readings r Reading: Disks, disk scheduling (3.7 of textbook; “How Stuff Works”) r Reading: File System Implementation ( of textbook)
David N. Brown Lawrence Berkeley National Lab Representing the BaBar Collaboration The BaBar Mini  BaBar  BaBar’s Data Formats  Design of the Mini 
CHEP Sep Andrey PHENIX Job Submission/Monitoring in transition to the Grid Infrastructure Andrey Y. Shevel, Barbara Jacak,
9 February 2000CHEP2000 Paper 3681 CDF Data Handling: Resource Management and Tests E.Buckley-Geer, S.Lammel, F.Ratnikov, T.Watts Hardware and Resources.
5 May 98 1 Jürgen Knobloch Computing Planning for ATLAS ATLAS Software Week 5 May 1998 Jürgen Knobloch Slides also on:
Fabiola Gianotti, 31/8/’99 PHYSICS and SOFTWARE ATLAS Software Week 31/8/’99 Fabiola Gianotti Software requirements of physics groups What should Detector.
PHENIX and the data grid >400 collaborators Active on 3 continents + Brazil 100’s of TB of data per year Complex data with multiple disparate physics goals.
EGEE is a project funded by the European Union under contract IST HEP Use Cases for Grid Computing J. A. Templon Undecided (NIKHEF) Grid Tutorial,
DPDs and Trigger Plans for Derived Physics Data Follow up and trigger specific issues Ricardo Gonçalo and Fabrizio Salvatore RHUL.
October 8, 2002P. Nilsson, SPD General Meeting1 Paul Nilsson, SPD General Meeting, Oct. 8, 2002 New tools and software updates Test beam analysis Software.
The LHCb Italian Tier-2 Domenico Galli, Bologna INFN CSN1 Roma,
9-13/9/03 Atlas Overview WeekPeter Sherwood 1 Atlfast, Artemis and Atlantis What, Where and How.
Marco Cattaneo, Aleph plenary, 23rd April Long term archive of LEP data  LEPC working group report Purpose Assumptions Conclusions  Physics goals.
Marco Cattaneo, 15-Sep OO software plans  Major milestone (presented last June) Fully functional SICB replacement by mid-2000  How to get there?
Introduction What is detector simulation? A detector simulation program must provide the possibility of describing accurately an experimental setup (both.
2-Dec Offline Report Matthias Schröder Topics: Scientific Linux Fatmen Monte Carlo Production.
Simulation Commissioning, Validation, Data Quality A brain dump to prompt discussion Many points applicable to any of LHCb software but some simulation.
Monte-Carlo Event Database: current status Sergey Belov, JINR, Dubna.
CBM ECAL simulation status Prokudin Mikhail ITEP.
Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,
A.Abhari CPS1251 Topic 1: Introduction to Computers Computer Hardware Computer components Connecting Computers Computer Software Operating System (OS)
UTA MC Production Farm & Grid Computing Activities Jae Yu UT Arlington DØRACE Workshop Feb. 12, 2002 UTA DØMC Farm MCFARM Job control and packaging software.
Claudio Grandi INFN-Bologna CHEP 2000Abstract B 029 Object Oriented simulation of the Level 1 Trigger system of a CMS muon chamber Claudio Grandi INFN-Bologna.
The MEG Offline Project General Architecture Offline Organization Responsibilities Milestones PSI 2/7/2004Corrado Gatto INFN.
LHCbComputing Computing for the LHCb Upgrade. 2 LHCb Upgrade: goal and timescale m LHCb upgrade will be operational after LS2 (~2020) m Increase significantly.
Preservation of LEP Data There is still hope Is there? Marcello Maggi, Ulrich Schwickerath, Matthias Schröder, , DPHEP7 1.
UK LVL1 Meeting, RAL, 31/01/00Alan Watson 1 ATLAS Trigger Simulations Present & Future? What tools exist? What are they good for? What are the limitations?
Proposal for the MEG Offline System Assisi 9/21/2004Corrado Gatto General Architecture Computing Model Organization & Responsibilities Milestones.
Status of Hall C 6 GeV Analysis Software Robust Fortran/CERNLIB code, “ENGINE”, for analysis of HMS/SOS coincidence and single arm experiments that has.
AliRoot survey: Reconstruction P.Hristov 11/06/2013.
Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.
2-December Offline Report Matthias Schröder Topics: Monte Carlo Production New Linux Version Tape Handling Desktop Computers.
MONTE CARLO TRANSPORT SIMULATION Panda Computing Week 2012, Torino.
Jianming Qian, UM/DØ Software & Computing Where we are now Where we want to go Overview Director’s Review, June 5, 2002.
CDF SAM Deployment Status Doug Benjamin Duke University (for the CDF Data Handling Group)
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
Analysis Model Zhengyun You University of California Irvine Mu2e Computing Review March 5-6, 2015 Mu2e-doc-5227.
1 Calice TB Review DESY 15/6/06D.R. Ward David Ward Post mortem on May’06 DESY running. What’s still needed for DESY analysis? What’s needed for CERN data.
LEP DATA PRESERVATION 11 years of data taking 4 Experiments Large Luminosity ~1200 Scientific Papers ALEPH Raw data 5 Terabytes DST 800 Gigabytes Mini.
HYDRA Framework. Setup of software environment Setup of software environment Using the documentation Using the documentation How to compile a program.
EOSCpilot WP4: Use Case 5 Material for
Migration of reconstruction and analysis software to C++
ALICE analysis preservation
Status of Brunel team and next steps
Simulation use cases for T2 in ALICE
LCG Monte-Carlo Events Data Base: current status and plans
ALICE Computing Upgrade Predrag Buncic
Preparations for the CMS-HI Computing Workshop in Bologna
Operating Systems Tasks 17/02/2019.
Modern PC operating systems
Agenda SICb Session Status of SICb software migration F.Ranjard
BES III Software: Short-term Plan ( )
Presentation transcript:

Status of OPAL data Matthias Schröder

About OPAL OPAL: The Omni Purpose Apparatus at LEP See OPAL Page for Particle PhysicistsOPAL Page for Particle Physicists Data taking from Jan-2009Status of OPAL data2

Data Types Raw Data – RAWD Data Summary Tapes – DDST, DST, CSDST File sizes: < 200 MB With very few exceptions (N-tuples) – No central management 27-Jan-20093Status of OPAL data

Raw Data Packed hit-level information In ZEBRA format Stored on tape – Maintained (copy to new media) by CERN/IT One copy on magneto-optical disks – Medium very good for archiving – Drives and cataloging an issue 27-Jan-20094Status of OPAL data

Data Summary Tapes Reconstructed tracks, clusters, event shape,… – Compressed Results of event selections In ZEBRA format – Event header for analysis pre-selection – No streaming, but direct access lists Subtypes – DDST: Hit info kept for all event types – CSDST: Hit info kept for some event types This is what users read for the analysis On tape (castor) – Intermixed with other users data 27-Jan-20095Status of OPAL data

OPAL Software F77 based Source in patchy – On purpose Heavily relying on CERNLIB – ZEBRA – Patchy – HBOOK/PAW Stored in afs – Copy in pubarch 27-Jan-20096Status of OPAL data

Monte Carlo Several Generators used – Output 4-vectors stored (in weird format) on tape GEANT3 for detector simulation Production system – Input 4-vectors, output reconstr. DDST Frequently used samples: another step makes CSDST – Made heavy use of fatmen – Not working anymore 27-Jan-20097Status of OPAL data

Data Preservation Objectives Allow to continue analysis activities Combined analyses with other experiments Check new results or models – Does require ability to produce MC Open Acces not foreseen – Knowledge of strenghts and weaknesses required – All future analysis accompanied by LTEB member 27-Jan-20098Status of OPAL data

Data Preservation Data: count on CERN/IT for tape migration – Preserve 4-vectors, RAWD, DDST, CSDST – Should have a second copy elsewhere – Catalog: shall we rely on castor only? No central effort to preserve Ntuples Software: try to keep alive – Migrate to new OS versions – Migrate to changing environments (castor2) – But rely on CERNLIB And availability of 32 bit compilers 27-Jan-20099Status of OPAL data

Data Preservation, ctd. Documentation – OPAL primer – Technical Notes – Wiki for documenting changes Just started Expertise – Long Term Editorial Board (LTEB) Experts from sub-detectors, computing, analysis WGs 27-Jan Status of OPAL data

Issues Data losses – Tapes do wear out Effects all types of data Technology changes – Compilers – Cernlib – Tape access – Catalog 27-Jan Status of OPAL data

Issues, ctd. Loss of software – Most analysis jobs stored in private directory – Problem with commercial software No more visualisation – No CERNLIB build for SL5 foreseen Loss of documentation – Private notes not collected centrally Loss of expertise – People retire, go to industry – People forget Accounts get deleted – Not every detail is documented Why write down the obvious… 27-Jan Status of OPAL data

Issues, ctd Resources – No more budgets – No more staff 27-Jan-2009Status of OPAL data13

Conclusion Started (too?) late Efforts to improve documentation (So far) manage to keep data alive Worries – Future of CERNLIB – Tapes wearing out – Fading memories 27-Jan-2009Status of OPAL data14