1. Overview of LHCb applications and software environment
LHCb software tutorial - September 2011

2. Event Model / Detector Description / Conditions
LHCb applications
Event Model / Detector Description / Conditions
MicroDST
GenParticles
Raw Data
Stripping
DaVinci
Analysis
DaVinci
Simul.
Gauss
Digit.
Boole
Analysis
Bender
HLT
Moore
Reco.
Brunel
DST
MCHits
Ev.Disp.
Panoramix
MCParticles
Gaudi
LHCb software tutorial - September 2011

3. Software organisation
Applications are released as a stack of Projects
One project per application
e.g. Brunel, DaVinci
Several independent projects for components
e.g. Lbcom, Rec, Hlt, Analysis, Stripping
Four projects for the framework
Gaudi, LHCb, Phys, Online
Projects are collections of Packages
Group of classes in a logically cohesive physical unit.
Packaging structure reflects on:
Logical structure of the application
Organizational structure of the development team
Package is minimal entity that can be versioned
A project version uniquely defines the versions of the packages it contains, and of the projects it depends on
Users work in the environment defined for a given version of the chosen project
e.g. SetupProject DaVinci v29r0
Brunel
Lbcom
Rec
LHCb
Gaudi
LHCb software tutorial - September 2011

4. CMT (Configuration Management Tool)
Based around the notion of package
Provides a set of tools for automating the configuration and building of packages
CMT
requirements
code
SVN
repository
What to build
How to build
Package dependencies
makefile
DevStudio files
Building
tools
(compilers,
linkers,
IDEs )
Libraries
&
Executables
CMT introduction
CMT is an attempt to formalize software production and especially configuration management around a package-oriented principle. The notion of packages represents hereafter a set of software components (that may be applications, libraries, documents, tools etc...) that are to be used for producing a system or a framework. In such an environment, several persons are assumed to participate in the development and the components themselves are either independent or related to each other.
The environment provides conventions (for naming packages, files, directories and for addressing them) and tools for automating as much as possible the implementation of these conventions. It permits to describe the configuration requirements and automatically deduce from the description the effective set of configuration parameters needed to operate the packages (typically for building them or using them).
LHCb software tutorial - September 2011

5. Package: Structure
Mandatory documentation directory contains the release.notes file
Hat
public scripts and configurables, will be on
$PYTHONPATH
packA
$PACKAROOT
cmt
options
python
doc
src
packA
Job options
Mandatory manager directory contains the requirements file
public include files
#include “packA/xxx.h”
Package Structure
All packages in LHCb follow the structure shown in this slide.
The /cmt directory is mandatory (see later with CMT)
The source files are located in /src and the exported header files are in /packA. In that way it is visible from which package a header is included.
The /doc directory contains the documentation for the package. It is mandatory to have a file called “release.notes” where we keep the changes on the package up to date.
A number of binary (platform & configuration dependent) directories will exists in each package.
Package Groups (Hats)
We have organized the packages in LHCb by putting together all related packages into a package group. Examples are: Event, Det, Ex, etc.
i686_winxp _vc9_dbg
x86_64_slc5 _icc11_opt
. . .
x86_64-slc5
-gcc43-opt
Generated binary directories
LHCb software tutorial - September 2011

6. Package versions Packages have several versions v28r1p1 Major version
Defined in cmt requirements file
Version number formatted according to convention:
v28r1p1
Major version
Indicates a change in the interface: all packages that use it may have to change
Minor version
Indicates an internal only change
Patch version
Not always present. A minor bug fix to an existing release
Major version 28
Minor version 1
Patch version 1
LHCb software tutorial - September 2011

7. Package Categories
Application: is a package that sets up the environment and contains the job options needed for configuring and running a program.
Library: contains a list of classes and the list of dependent packages needed to compile it.
Package group: contains a list of other packages with their version number (e.g. LHCbSys)
Interface package: interfacing to packages not managed with CMT (e.g. Python, GSL, ROOT,…)
Package Categories
With respect to CMT it is interesting to distinguish the different categories of packages. They are used in the exactly the same way but their requirement files will show some differences, specially in patterns that are used.
The concept of interface package is interesting for integrating in the build system packages that have been developed outside CMT. Basically these packages only define a number of macros and environment variables needed for compiling, linking and running with these external packages. These interfaces packages are defined currently in the LCGCMT project (common between the various LHC experiments using CMT)
Package group are useful for fixing a set on compatible versions of other packages. In this case the package using this set of packages needs only to state the version number of the package group.
LHCb software tutorial - September 2011

8. Link vs. Component Libraries
Link libraries are linked to the program executable
(static or dynamic linking)
Base classes, event and geometry data classes
Loaded at program start
Must be found on LD_LIBRARY_PATH
Component libraries are loaded on demand at run-time
Collection of components (Algorithms, Tools, Services, etc.)
“Plug-ins”
No need to recompile or re-link program if plug-in changes
Link libraries
These are traditional libraries that, as the name implies, will be linked into the application. Typically they contain base classes or data objects.
Component Libraries
Component libraries are shared libraries that contain standard framework components which implement abstract interfaces. Such components are Algorithms, Auditors, Services, Tools and Converters. These libraries do not export their symbols apart from the one which is used by the framework to discover what components are contained in the library. These libraries are loaded at run time – this means that you do not have to relink the application if you rebuild the library
N.B. <Components>_dll.cpp
This file was needed in the past for technical reasons, but it’s no longer needed.
The Tutorial will be based on the development of a “component” library that will include all the Algorithms that we are going to develop during the practical exercises.
LHCb software tutorial - September 2011

9. CMT: requirements file
package MyPackage
version v1r0
# Structure, i.e. directories to process.
branches cmt doc src
# Used packages.
use GaudiAlg v*
# Component library building rule
library MyPackage ../src/*.cpp
# define component library link options
apply_pattern component_library library=MyPackage
Some basic keywords
package. Defines the name of the package
version. Defines the version of the package. The number after the “v” is the major version number, that after the “r” the minor version number (“p” is also available, for patches). We increase the major version if there are major changes in functionality that are likely to affect other packages (e.g. modified interfaces, incompatible changes in data objects, changed behaviour of algorithms), the minor version otherwise (e.g. new classes added, backward compatible bug fixes)
use. Instructs CMT on the dependencies of this package to other CMT packages. Normally the wildcard “*” is sufficient because the specific versions are frozen by the project. Explicit major versions can be a useful indication when integrating the software that the package can only work with a specific version of a dependent package. The name that sometimes appears after the version is the location of the package (package group).
library. Tells CMT that this package is constituted of a program (application) and where to locate the sources. Similarly application
apply_pattern. Tells CMT to apply one or more CMT commands (typically a macro definition) following a pattern that is common to all packages of a given type (in this case applications). Other common patterns are linker_library and application_path.
LHCb software tutorial - September 2011

10. CMT: Basic Commands
All commands must be executed from a directory containing a cmt requirements file
cmt config
Configures the package (creates setup and make files)
Invoked automatically by getpack (see later slide)
cmt show uses and/or cmt show projects
Show dependencies and actual versions used
cmt make
build the current package
cmt broadcast <command>
Recursive CMT command in all used packages found in the current CMT project
e.g. cmt broadcast cmt make
cmt run <command>
Executes <command> in current package environment
CMT Primary commands
configure a package
> cd cmtuser/package/cmt
> cmt config
visualize packages and version numbers used by a library or an application.
> cmt show uses
get macro definitions used in makefiles
> cmt show macros
get one specific macro used in makefiles
> cmt show macro cppflags
LHCb software tutorial - September 2011

11. Setting the CMT environment
Create a working directory for a given project
actually an alias for SetupProject --build-env <Project> [<version>]
creates working directory and cd to it: ~/cmtuser/<Project>_<version>
Packages are searched for in Projects that are on CMTPROJECTPATH
Default is ${User_release_area}:${LHCBPROJECTPATH}
Add additional paths with --nightly and --dev-dir switches of SetupProject
> setenv<Project> [<version>]
CMT and projects
CMTPROJECTPATH Is the list of top directories to look for CMT projects. It is set by the setenv<Project> and SetupProject scripts
CMTCONFIG Is the default configuration used by the CMT commands. It is set by default at login time to the current platform name and recommended default compiler (e.g. x86_64-slc5-gcc43-opt). The User can change it to a different default (e.g. x86_64-slc5-gcc43-dbg to turn on the debug configuration)
LHCb software tutorial - September 2011

12. Getting a package The “getpack” command
Script combining “svn checkout” + “cmt config”
If no version given, it suggests the latest version of the package consistent with the current environment
Suggested version is not necessarily the most recent version if you are not using the latest environment
> getpack [hat/]<package> [<version>] [head]
Getting a package
The “getpack” command has been developed to ease the use for getting a copy of a package from the LHCb SVN repository and putting it in the correct directory structure with the correct version. It also suggests to the user what versions are available for the package in case the user does not specify it.
For the tutorial we recommend to get always the “head” revision of the packages.
LHCb software tutorial - September 2011

13. Building a package Work in the /cmt directory
<hat>/<package>/cmt
Set the CMT configuration ($CMTCONFIG)
Defines platform, compiler, directories where to find binaries
slc5 default: x86_64-slc5-gcc43-opt
LbLogin –c i686-slc5-gcc43-dbg
setenv CMTCONFIG $CMTDEB
Invoke the make command, via cmt
In general, putting “cmt run” in front of any command ensures that the command is executed in a shell that has defined all the environment described by the requirements file. “cmt make” is a special case.
> cmt make [-j][target][clean][binclean]
Building a package
Typically for building (and also running) a package we stay in the /cmt directory. This is convenient for executing the configuration and build commands.
The CMTCONFIG environment variable
Tells CMT which configuration (platform+compiler+compilation options) combination to build for. The default is set at login time.
The make command. The “make” command is used without arguments for building the libraries and/or the executables. The binary files are stored in the $CMTCONFIG directory and copied to the project InstallArea. Note that “make” and “gmake” are equivalent on lxplus. Depending on the complexity of the requirements file it can also be invoked with a [target] argument to only do a partial build. When invoked with the “clean” argument, the command undoes all build actions. This can be slow because CMT recomputes all dependencies etc. - If you just want to rebuild unconditionally from sources, use the “binclean” argument, which simply deletes the binary directory and all libraries it contains.
Putting “cmt “ in front of any command (e.g. cmt make) ensures that the command is executed inside the environment described by the cmt requirements file in the current directory
LHCb software tutorial - September 2011

14. Building several packages
A handy Makefile that wraps cmt is available in the directory <Project>_<version>
Allows to:
build all the packages in the directory
build a package with the ones it depends on
build just one package
Example workflow:
> make
> make <hat>/<package>
> make PKG_ONLY=1 <hat>/<package>
> SetupProject --build-env MyProject
> getpack Pack1
> getpack Pack2
> make
LHCb software tutorial - September 2011

15. Running the application
Set the run time environment
Takes into account value of CMTCONFIG
Needed once after the build, then only if environment changes (new packages checked out, requirements file changed, CMTCONFIG changed, etc.)
Execute the program
But see also cmt run command in hands on
> SetupProject <project> <version>
gaudirun.py job.py
LHCb software tutorial - September 2011

16. SVN (Version Control System)
Record the history of your source files
Share your development with other people
Commit often
But only codes that compiles and builds
Tested by nightly builds
LHCb and Gaudi Repositories reside on CERN-IT SVN server
Private repositories are also possible, but must be managed by yourself, see requesting
Basic Description of SVN
SVN is a system that lets groups of people work simultaneously on groups of files (for instance packages).
It works by holding a central `repository' of the most recent version of the files. You may at any time create a personal copy of these files by `checking out' the files from the repository into one of your directories. If at a later date newer versions of the files are put in the repository, you can `update' your copy.
You may edit your copy of the files freely. If new versions of the files have been put in the repository in the meantime, doing an update merges the changes in the central copy into your copy.
When you are satisfied with the changes you have made in your copy of the files, you can `commit' them into the central repository.
When you are finally done with your personal copy of the files, you can `release' them and then remove them
LHCb software tutorial - September 2011

17. Accessing LHCb SVN server
Web browsable
LHCb:
Trac:
WebSVN:
Gaudi:
Trac:
WebSVN:
World readable if authenticated
SSH authentication
Automatic if authenticated in CERN AFS cell (lxplus)
For write access
Register your account in the e-group lhcb-svn-writers
Detailed instructions at
LHCb software tutorial - September 2011

18. Eclipse IDE customisation
Eclipse plugins to help LHCb developers
Wizards for
SetupProject --build-env
getpack
Creation of requirements, components (Algoriithms, Tools, ...)
Shared pre-configured installation on AFS (used in the hands- on)
Eclipse repository of packages (update site) for easy installation of useful plugins
Documentation on Twiki
LHCb software tutorial - September 2011

19. Emacs customisation A customisation of emacs for LHCb:
Templates for creation of files
E.g. MyAlgorithm.h, MyAlgorithm.cpp, requirements, release.notes etc.
Various shortcuts for code insertions
Optionally, load an EDT keypad emulation
Add following lines to ~/.emacs:
(load (expand-file-name "$EMACSDIR/edt"))
(load (expand-file-name "$EMACSDIR/lhcb"))
Or copy from $EMACSDIR/.emacs
LHCb emacs customisation
Documentation at:
LHCb software tutorial - September 2011

20. Exercise
Now read the web page attached to this lesson in the agenda and work through the exercises
LHCb software tutorial - September 2011