Overview of LHCb applications and software environment LHCb software tutorial - September 2011

Event Model / Detector Description / Conditions LHCb applications Event Model / Detector Description / Conditions MicroDST GenParticles Raw Data Stripping DaVinci Analysis DaVinci Simul. Gauss Digit. Boole Analysis Bender HLT Moore Reco. Brunel DST MCHits Ev.Disp. Panoramix MCParticles Gaudi LHCb software tutorial - September 2011

Software organisation Applications are released as a stack of Projects One project per application e.g. Brunel, DaVinci Several independent projects for components e.g. Lbcom, Rec, Hlt, Analysis, Stripping Four projects for the framework Gaudi, LHCb, Phys, Online Projects are collections of Packages Group of classes in a logically cohesive physical unit. Packaging structure reflects on: Logical structure of the application Organizational structure of the development team Package is minimal entity that can be versioned A project version uniquely defines the versions of the packages it contains, and of the projects it depends on Users work in the environment defined for a given version of the chosen project e.g. SetupProject DaVinci v29r0 Brunel Lbcom Rec LHCb Gaudi LHCb software tutorial - September 2011

CMT (Configuration Management Tool) Based around the notion of package Provides a set of tools for automating the configuration and building of packages CMT requirements code SVN repository What to build How to build Package dependencies makefile DevStudio files Building tools (compilers, linkers, IDEs ) Libraries & Executables CMT introduction CMT is an attempt to formalize software production and especially configuration management around a package-oriented principle. The notion of packages represents hereafter a set of software components (that may be applications, libraries, documents, tools etc...) that are to be used for producing a system or a framework. In such an environment, several persons are assumed to participate in the development and the components themselves are either independent or related to each other. The environment provides conventions (for naming packages, files, directories and for addressing them) and tools for automating as much as possible the implementation of these conventions. It permits to describe the configuration requirements and automatically deduce from the description the effective set of configuration parameters needed to operate the packages (typically for building them or using them). LHCb software tutorial - September 2011

Package: Structure Mandatory documentation directory contains the release.notes file Hat public scripts and configurables, will be on $PYTHONPATH packA $PACKAROOT cmt options python doc src packA Job options Mandatory manager directory contains the requirements file public include files #include “packA/xxx.h” Package Structure All packages in LHCb follow the structure shown in this slide. The /cmt directory is mandatory (see later with CMT) The source files are located in /src and the exported header files are in /packA. In that way it is visible from which package a header is included. The /doc directory contains the documentation for the package. It is mandatory to have a file called “release.notes” where we keep the changes on the package up to date. A number of binary (platform & configuration dependent) directories will exists in each package. Package Groups (Hats) We have organized the packages in LHCb by putting together all related packages into a package group. Examples are: Event, Det, Ex, etc. i686_winxp _vc9_dbg x86_64_slc5 _icc11_opt . . . x86_64-slc5 -gcc43-opt Generated binary directories LHCb software tutorial - September 2011

Package versions Packages have several versions v28r1p1 Major version Defined in cmt requirements file Version number formatted according to convention: v28r1p1 Major version Indicates a change in the interface: all packages that use it may have to change Minor version Indicates an internal only change Patch version Not always present. A minor bug fix to an existing release Major version 28 Minor version 1 Patch version 1 LHCb software tutorial - September 2011

Package Categories Application: is a package that sets up the environment and contains the job options needed for configuring and running a program. Library: contains a list of classes and the list of dependent packages needed to compile it. Package group: contains a list of other packages with their version number (e.g. LHCbSys) Interface package: interfacing to packages not managed with CMT (e.g. Python, GSL, ROOT,…) Package Categories With respect to CMT it is interesting to distinguish the different categories of packages. They are used in the exactly the same way but their requirement files will show some differences, specially in patterns that are used. The concept of interface package is interesting for integrating in the build system packages that have been developed outside CMT. Basically these packages only define a number of macros and environment variables needed for compiling, linking and running with these external packages. These interfaces packages are defined currently in the LCGCMT project (common between the various LHC experiments using CMT) Package group are useful for fixing a set on compatible versions of other packages. In this case the package using this set of packages needs only to state the version number of the package group. LHCb software tutorial - September 2011

Link vs. Component Libraries Link libraries are linked to the program executable (static or dynamic linking) Base classes, event and geometry data classes Loaded at program start Must be found on LD_LIBRARY_PATH Component libraries are loaded on demand at run-time Collection of components (Algorithms, Tools, Services, etc.) “Plug-ins” No need to recompile or re-link program if plug-in changes Link libraries These are traditional libraries that, as the name implies, will be linked into the application. Typically they contain base classes or data objects. Component Libraries Component libraries are shared libraries that contain standard framework components which implement abstract interfaces. Such components are Algorithms, Auditors, Services, Tools and Converters. These libraries do not export their symbols apart from the one which is used by the framework to discover what components are contained in the library. These libraries are loaded at run time – this means that you do not have to relink the application if you rebuild the library N.B. <Components>_dll.cpp This file was needed in the past for technical reasons, but it’s no longer needed. The Tutorial will be based on the development of a “component” library that will include all the Algorithms that we are going to develop during the practical exercises. LHCb software tutorial - September 2011

CMT: requirements file package MyPackage version v1r0 # Structure, i.e. directories to process. branches cmt doc src # Used packages. use GaudiAlg v* # Component library building rule library MyPackage ../src/*.cpp # define component library link options apply_pattern component_library library=MyPackage Some basic keywords package. Defines the name of the package version. Defines the version of the package. The number after the “v” is the major version number, that after the “r” the minor version number (“p” is also available, for patches). We increase the major version if there are major changes in functionality that are likely to affect other packages (e.g. modified interfaces, incompatible changes in data objects, changed behaviour of algorithms), the minor version otherwise (e.g. new classes added, backward compatible bug fixes) use. Instructs CMT on the dependencies of this package to other CMT packages. Normally the wildcard “*” is sufficient because the specific versions are frozen by the project. Explicit major versions can be a useful indication when integrating the software that the package can only work with a specific version of a dependent package. The name that sometimes appears after the version is the location of the package (package group). library. Tells CMT that this package is constituted of a program (application) and where to locate the sources. Similarly application apply_pattern. Tells CMT to apply one or more CMT commands (typically a macro definition) following a pattern that is common to all packages of a given type (in this case applications). Other common patterns are linker_library and application_path. LHCb software tutorial - September 2011

CMT: Basic Commands All commands must be executed from a directory containing a cmt requirements file cmt config Configures the package (creates setup and make files) Invoked automatically by getpack (see later slide) cmt show uses and/or cmt show projects Show dependencies and actual versions used cmt make build the current package cmt broadcast <command> Recursive CMT command in all used packages found in the current CMT project e.g. cmt broadcast cmt make cmt run <command> Executes <command> in current package environment CMT Primary commands configure a package > cd cmtuser/package/cmt > cmt config visualize packages and version numbers used by a library or an application. > cmt show uses get macro definitions used in makefiles > cmt show macros get one specific macro used in makefiles > cmt show macro cppflags LHCb software tutorial - September 2011

Setting the CMT environment Create a working directory for a given project actually an alias for SetupProject --build-env <Project> [<version>] creates working directory and cd to it: ~/cmtuser/<Project>_<version> Packages are searched for in Projects that are on CMTPROJECTPATH Default is ${User_release_area}:${LHCBPROJECTPATH} Add additional paths with --nightly and --dev-dir switches of SetupProject > setenv<Project> [<version>] CMT and projects CMTPROJECTPATH Is the list of top directories to look for CMT projects. It is set by the setenv<Project> and SetupProject scripts CMTCONFIG Is the default configuration used by the CMT commands. It is set by default at login time to the current platform name and recommended default compiler (e.g. x86_64-slc5-gcc43-opt). The User can change it to a different default (e.g. x86_64-slc5-gcc43-dbg to turn on the debug configuration) LHCb software tutorial - September 2011

Getting a package The “getpack” command Script combining “svn checkout” + “cmt config” If no version given, it suggests the latest version of the package consistent with the current environment Suggested version is not necessarily the most recent version if you are not using the latest environment > getpack [hat/]<package> [<version>] [head] Getting a package The “getpack” command has been developed to ease the use for getting a copy of a package from the LHCb SVN repository and putting it in the correct directory structure with the correct version. It also suggests to the user what versions are available for the package in case the user does not specify it. For the tutorial we recommend to get always the “head” revision of the packages. LHCb software tutorial - September 2011

Building a package Work in the /cmt directory <hat>/<package>/cmt Set the CMT configuration ($CMTCONFIG) Defines platform, compiler, directories where to find binaries slc5 default: x86_64-slc5-gcc43-opt LbLogin –c i686-slc5-gcc43-dbg setenv CMTCONFIG $CMTDEB Invoke the make command, via cmt In general, putting “cmt run” in front of any command ensures that the command is executed in a shell that has defined all the environment described by the requirements file. “cmt make” is a special case. > cmt make [-j][target][clean][binclean] Building a package Typically for building (and also running) a package we stay in the /cmt directory. This is convenient for executing the configuration and build commands. The CMTCONFIG environment variable Tells CMT which configuration (platform+compiler+compilation options) combination to build for. The default is set at login time. The make command. The “make” command is used without arguments for building the libraries and/or the executables. The binary files are stored in the $CMTCONFIG directory and copied to the project InstallArea. Note that “make” and “gmake” are equivalent on lxplus. Depending on the complexity of the requirements file it can also be invoked with a [target] argument to only do a partial build. When invoked with the “clean” argument, the command undoes all build actions. This can be slow because CMT recomputes all dependencies etc. - If you just want to rebuild unconditionally from sources, use the “binclean” argument, which simply deletes the binary directory and all libraries it contains. Putting “cmt “ in front of any command (e.g. cmt make) ensures that the command is executed inside the environment described by the cmt requirements file in the current directory LHCb software tutorial - September 2011

Building several packages A handy Makefile that wraps cmt is available in the directory <Project>_<version> Allows to: build all the packages in the directory build a package with the ones it depends on build just one package Example workflow: > make > make <hat>/<package> > make PKG_ONLY=1 <hat>/<package> > SetupProject --build-env MyProject > getpack Pack1 > getpack Pack2 > make LHCb software tutorial - September 2011

Running the application Set the run time environment Takes into account value of CMTCONFIG Needed once after the build, then only if environment changes (new packages checked out, requirements file changed, CMTCONFIG changed, etc.) Execute the program But see also cmt run command in hands on > SetupProject <project> <version> gaudirun.py job.py LHCb software tutorial - September 2011

SVN (Version Control System) Record the history of your source files Share your development with other people Commit often But only codes that compiles and builds Tested by nightly builds LHCb and Gaudi Repositories reside on CERN-IT SVN server Private repositories are also possible, but must be managed by yourself, see http://svn.web.cern.ch/svn/howto.php#creating- requesting Basic Description of SVN SVN is a system that lets groups of people work simultaneously on groups of files (for instance packages). It works by holding a central `repository' of the most recent version of the files. You may at any time create a personal copy of these files by `checking out' the files from the repository into one of your directories. If at a later date newer versions of the files are put in the repository, you can `update' your copy. You may edit your copy of the files freely. If new versions of the files have been put in the repository in the meantime, doing an update merges the changes in the central copy into your copy. When you are satisfied with the changes you have made in your copy of the files, you can `commit' them into the central repository. When you are finally done with your personal copy of the files, you can `release' them and then remove them LHCb software tutorial - September 2011

Accessing LHCb SVN server Web browsable LHCb: Trac: https://svnweb.cern.ch/trac/lhcb/browser WebSVN: http://svnweb.cern.ch/world/wsvn/lhcb Gaudi: Trac: https://svnweb.cern.ch/trac/gaudi/browser WebSVN: http://svnweb.cern.ch/world/wsvn/gaudi World readable if authenticated SSH authentication Automatic if authenticated in CERN AFS cell (lxplus) For write access Register your account in the e-group lhcb-svn-writers Detailed instructions at https://twiki.cern.ch/twiki/bin/view/LHCb/SVNUsageGuidelines LHCb software tutorial - September 2011

Eclipse IDE customisation Eclipse plugins to help LHCb developers Wizards for SetupProject --build-env getpack Creation of requirements, components (Algoriithms, Tools, ...) Shared pre-configured installation on AFS (used in the hands- on) Eclipse repository of packages (update site) for easy installation of useful plugins Documentation on Twiki https://twiki.cern.ch/twiki/bin/view/LHCb/EclipseConfiguration https://twiki.cern.ch/twiki/bin/view/LHCb/EclipseTutorial https://twiki.cern.ch/twiki/bin/view/LHCb/EclipsePluginsTutorial LHCb software tutorial - September 2011

Emacs customisation A customisation of emacs for LHCb: Templates for creation of files E.g. MyAlgorithm.h, MyAlgorithm.cpp, requirements, release.notes etc. Various shortcuts for code insertions Optionally, load an EDT keypad emulation Add following lines to ~/.emacs: (load (expand-file-name "$EMACSDIR/edt")) (load (expand-file-name "$EMACSDIR/lhcb")) Or copy from $EMACSDIR/.emacs LHCb emacs customisation Documentation at: http://lhcb-comp.web.cern.ch/lhcb-comp/Support/emacs.htm LHCb software tutorial - September 2011

Exercise Now read the web page attached to this lesson in the agenda and work through the exercises LHCb software tutorial - September 2011