Separate distribution of the analysis code (and more) P. Hristov 19/03/2014.

Slides:



Advertisements
Similar presentations
Simple Git Steve Pieper. Topics Git considerations and Slicer Git as if it were svn Git the way it is meant to be.
Advertisements

Software Configuration Management Donna Albino LIS489, December 3, 2014.
The LEGO Train Framework
Presented by IBM developer Works ibm.com/developerworks/ 2006 January – April © 2006 IBM Corporation. Making the most of Creating Eclipse plug-ins.
Chapter 25 – Configuration Management 1Chapter 25 Configuration management.
How to install CGAL Yuanzhen Wang. What is CGAL Computational Geometry Algorithms Library “Provide easy access to efficient and reliable geometric algorithms.
Linux Operations and Administration
SCRAM Software Configuration, Release And Management Background SCRAM has been developed to enable large, geographically dispersed and autonomous groups.
Tools and software process for the FLP prototype B. von Haller 9. June 2015 CERN.
SUSE Linux Enterprise Server Administration (Course 3037) Chapter 4 Manage Software for SUSE Linux Enterprise Server.
 To explain the importance of software configuration management (CM)  To describe key CM activities namely CM planning, change management, version management.
GIT An introduction to GIT Source Control. What is GIT (1 of 2) ▪ “Git is a free and open source distributed version control system designed to handle.
Usability Issues Documentation J. Apostolakis for Geant4 16 January 2009.
Configuration Management (CM)
Configuration Management Geant4 Review 19 June 2001 Gunter Folger.
Common Servers in a Workplace Environment Brandon Reynolds Computer Electronic Networking Dept. of Technology, Eastern Kentucky University.
Geant4 Installation Supported platforms:  Scientific Linux with gcc 4.1.2/4.6  Mac Os X 10.7 and 10.8 with gcc 4.21  Windows7 with Visual Studio.
Version Control Group Wen-hao Zeng Richard Liou. 2 Introduction Several groups develop the ITS concurrently Accumulated modification of files can cause.
Copyright © 2015 – Curt Hill Version Control Systems Why use? What systems? What functions?
Study for Migration from CVS to SubVersion (SVN) Gunter Folger CERN/PH/SFT.
1Offline Weekly Meeting May May 2006 AliRoot Build Integration and (Testing) System Peter Hristov Vagner Morais.
AliRoot Implementation of native CMake build system A. Grigoras, P. Hristov 19/11/20141Alice - CMake.
CMake refactoring P. Hristov 19/03/2014. History I  Recursive makefiles (F.Carminati):  Problems in dependencies  Slow  "Recursive Makefiles.
1 / 22 AliRoot and AliEn Build Integration and Testing System.
G4MICE Status and Plans 1M.Ellis - CM24 - RAL - 31st May 2009  Firstly, a correction to the agenda:  I failed to spot a mistake in the agenda that I.
Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G. Rybkine d, A. Undrus e University of Indiana, USA a,
Feedback from the POOL Project User Feedback from the POOL Project Dirk Düllmann, LCG-POOL LCG Application Area Internal Review October 2003.
LCG Generator Meeting, December 11 th 2003 Introduction to the LCG Generator Monthly Meeting.
Wordpress with Mina Automated Deployment Solution Jonathan Gravato DIG 4104c.
1 ALICE OFFLINE :ORGANIZATION A user point of view Yves Schutz, SUBATECH, France (PHOS biased)
Alex Undrus – Nightly Builds – ATLAS SW Week – Dec Preamble: Code Referencing Code Referencing is a vital service to cope with 7 million lines of.
AliRoot survey P.Hristov 11/06/2013. Offline framework  AliRoot in development since 1998  Directly based on ROOT  Used since the detector TDR’s for.
Introduction What is detector simulation? A detector simulation program must provide the possibility of describing accurately an experimental setup (both.
Feedback from LHC Experiments on using CLHEP Lorenzo Moneta CLHEP workshop 28 January 2003.
Servicing HEP experiments with a complete set of ready integrated and configured common software components Stefan Roiser 1, Ana Gaspar 1, Yves Perrin.
CERN IT Department t LHCb Software Distribution Roberto Santinelli CERN IT/GS.
Software Engineering Overview DTI International Technology Service-Global Watch Mission “Mission to CERN in Distributed IT Applications” June 2004.
A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Software Process & Infrastructure for LCG Project Overview LCG Application Area Internal.
Yannick Patois - Datagrid Software Repository Presentation - March, n° 1 Datagrid Software Repository Presentation CVS, packages and automatic.
Migration to ISO C++ an overview Gabriele Cosmo CERN - IT/ASD.
Software Deployment & Release 26/03/2015 1EN-ICE.
A. Gheata, ALICE offline week March 09 Status of the analysis framework.
Transition to SVN server: follow up P.Hristov 24/01/2008.
Data processing Offline review Feb 2, Productions, tools and results Three basic types of processing RAW MC Trains/AODs I will go through these.
SVN code server for AliRoot P.Hristov 18/10/2007.
Analysis train M.Gheata ALICE offline week, 17 March '09.
20 October 2005 LCG Generator Services monthly meeting, CERN Validation of GENSER & News on GENSER Alexander Toropin LCG Generator Services monthly meeting.
Maven. Introduction Using Maven (I) – Installing the Maven plugin for Eclipse – Creating a Maven Project – Building the Project Understanding the POM.
Some topics for discussion 31/03/2016 P. Hristov 1.
DECTRIS Ltd Baden-Daettwil Switzerland Continuous Integration and Automatic Testing for the FLUKA release using Jenkins (and Docker)
Chapter 25 – Configuration Management 1Chapter 25 Configuration management.
Computing in High Energy and Nuclear Physics 2012 May 21-25, 2012 New York United States The version control service for ATLAS data acquisition configuration.
Git Girish Git VCS that I have used ClearCase, cvs, svn Happy p4 user.
1 April 2, Software Packaging and Releasing Best Practices William Cohen NCSU CSC 591W April 2, 2008.
Build and Test system for FairRoot
Installation of the ALICE Software
CEPC Software Management Proposal
SPI external software build tool and distribution mechanism
Transition from git.cern.ch
ATLAS Software Distribution
Feedback on a first use of the ALFA and O2 software
Analysis Trains - Reloaded
Generator Services planning meeting
Releases and developments
Build and Package CSE 403 section, 2011 Feb 3, Hao Lu.
Cmake Primer.
Chapter 25 – Configuration Management
JENKINS TIPS Ideas for making your life with Jenkins easier
Beta releases and Product Management
Carthage ios 8 onwards Dependency manager that streamlines the process of integrating the libraries into the project.
Presentation transcript:

Separate distribution of the analysis code (and more) P. Hristov 19/03/2014

Reminder 2  After a lot of discussions in PWGs (and outside) Jan Fiete presented the proposal on 10/02/2014 presented  Motivation  Some parts of AliRoot change more frequently than others  AN tag often only for analysis code  Proposal  Split AliRoot into two parts  PWG = (PWG, PWGCF, PWGDQ, …): 9 folders in total = 0.3 GB  CORE = STEER, subdetectors, OCDB, OADB, ANALYSIS, etc.: i.e. everything else = 0.9 GB  Separate repositories  Separate tagging  CORE for example monthly, or on request  PWG twice weekly (like AN tags now)

Reminder: Tagging  Separate tagging, for example  CORE for example monthly, or on request  PWG twice weekly (like AN tags now) CORE v1 PWG v1 GEANT3 ROOT PWG v2 PWG v3 PWG v4 PWG v5 CORE v2 3

Reminder: Some Comments  Advantages  Stable CORE part  Regular PWG tags lead to faster build & smaller archive  Committers to PWG don't need to update CORE part on each commit  Implications for users  Need to download one package more  Detector developers may not need PWG package  Need to build one package more  May build into same directory, no change in includes, library path required  AliRoot download script can do this transparently  Implications for train operators  Selection of PWG tag instead of AliRoot tag, dependencies automatic  Decision: split the PWG code in a separate package after QM 4

Discussion 5  The main question is how we keep consistency between the packages  Other experiments use Code Management Tools: quite heavy  The versioning can be improved and the dependencies between the packages described in a way similar to what RPM and deb do  The solution should be extensible to the external packages (Geant3, Geant4, BOOST, CGAL, FASTJET, etc.)  We can use the FairRoot way of distribution

FairRoot distribution method 6  “Meta-repository” for the external packages + compilation macro (configure.sh). Example: dec13 release (or FairSoft from github)  Git  HepMC  Millepede-II V  ØMQ  CMAKE  BOOST  ROOT  Geant  Geant 3.21 v1-15  Geant4_VMC v2-14a  VGM v3-06  PYTHIA  GSL 1.15  GTEST  PLUTO 5.37  PYTHIA  Core package (FairRoot)  Experiment specific code, i.e. CbmRoot

Proposal for a new AliRoot distribution method 7  Meta-repository for the external packages + installation script  Links to the needed versions of the original distributions: repositories, tarballs, etc.  ROOT  Generators  Transport packages & VMC  Additional software  Patches for the original distributions  Distributions, not available in original form (i.e. DPMJET)  Interfaces to packages (i.e. THijing)  The versions in each release will be decided on a dedicated offline meeting taking into account the requests from the PWGs  Core package (STEER, detectors, ANALYSIS, etc.)  Analysis package (PWG, PWGXX)

Benefits 8  Clear set of versions for each release of external packages, core and analysis  Easy introduction of new external code  Clear set of patches (now they are on the build server, but rarely people take them)  Low/high frequency of changes => low/high release frequency  Smaller distribution size for the frequently updated code (PWG analysis)  Easier installation  Better access control and notification

Plan 9  Analysis packages (2-3 weeks)  Split git repositories  Split cmake files. Profit to implement the “native CMake build”, if not yet done  Adapt build server  Adapt train system  Adapt MonALISA scripts  Adapt AliRoot download scripts  Adapt documentation  External packages (2-3 weeks)  Prepare list of external packages with their versions, interfaces, patches, etc.  Create the meta-repository and the compilation script  Extensive tests on different platforms  Adapt the build system  Modify the versioning in CORE and introduce the requirements for the external packages  Exclude the existing external packages from CORE  Documentation  CORE (1-2 weeks)  Modifications in CMake build system and the servers