1. CMT http://www.cmtsite.org/ Define the development work models
Christian Arnault
Define the development
work models
Project management
Split the complete software base into natural sub-projects
Based on the release cycle constraints
Typical:
Core software
Offline/online
Simulation/reconstruction/analysis/…
Reflect the multi-project organization
Eg. Atlas uses Gaudi which uses LCGCMT
Decomposition based on the responsibilities
Each individual project have
A specific release cycle
Every n-week
Random
A dedicated CMTPATH entry
An interface package to its used sub-projects
Defines the upstream CMTPATH items
Selects the release id of all used sub-projects
Development/Integration work models
The work area becomes the bottom project
i.e. it is the top left entry in $CMTPATH
Used packages override those from the selected production release
Structure the software base with levels of integration areas
Structuring the packages
A package belongs to a project
It is located in the corresponding CMTPATH entry
May be located at the top position or below any directory tree
A package has a unique name
must be unique in the complete software base
Unique amongst the different projects
A package has versions
Implicitly specified in the directory structure
Or explicitly in the version.cmt text file
A package is recognized as a CMT package by
a cmt directory below its directory structure
a requirements file in the cmt directory
/.../Project/cmt/project.cmt
/A/cmt/requirements
/cmt/version.cmt
/B/v1/cmt/requirements
/C/D/E/cmt/requirements
/version.cmt
The project file
Package A
with explicit
version definition
Package B
with implicit
Package E
In a sub-directory
Structuring the projects
CMTPATH=/…/Project1 : /…/Project2 : /…/Project3
Project1
Project2
Project3
Projects are located in dedicated disk areas
Each project has an entry in the CMTPATH search list
Follows the standard platform dependent syntax
‘:’ on Unix and ‘;’ on Windows
The search list is prioritized like usual PATH search lists
Left (bottom) entries are considered first
Right (top) entries are considered last
Packages of a project may use packages of upstream projects
Packages belong to projects
they are always installed in one of the CMTPATH entries
This is where packages are looked for
Each project may define its own strategies
In /…/Project/cmt/project.cmt
Valid for all packages in this project
Valid for all bottom packages if not overridden
project P1
package B
use A
package C
use B
package D
package E
package G
use E
use F
project P2
package A
project P3
package F
Integration area
development area
Identify the elements
of the configuration
Projects
Project names are defined in the project files
<project>/cmt/project.cmt
Packages
The name is the primary identifier of packages
Must be unique
Constituents
Libraries
Applications
Generated documents
Activities
Code generation
Build
One make target per constituent
Actions
Generic activity
Activities can be grouped by type (testing, documentation, installation, deployment…)
Contextual parameters values according to
Sites
Platforms (hardware, OS, …)
Step in the Life cycle
User defined conditions
Specified in symbols
Macros, sets, paths, actions
Variants with tags or tag mixtures
Structure the projects
Structure the packages
Define the development work models
Define recurrent patterns
in configuration management
Identify the elements of the configuration
Query the knowledge base to
parameterize the development tools
Identify and describe the activities
of the software production
Define recurrent patterns
in configuration management
Build strategy
E.g. Define whether the result of the build will go to a global installation area
Setup strategy
E.g. Define what kind of automatic environment variables will be generated by CMT
Compiling, linking
E.g. Construct DLL on different platforms
Generating code, documentation
E.g. Building rootcint generator
Complec Doxygen generation with tag-file management
Include file search style
E.g. Specifying the project-wide conventions for include search strategy
Test patterns
Driving CppUnit
Enforcing the Unit tests
Managing the integration tests
QA activities
Query the knowledge base to
parameterize the development tools
The cmt command
A C++ application
Available on any platform (Unixes, MacOSX, native Windows)
All configuration parameters can be reached by query commands
cmt show macros
cmt show uses
cmt show macro xxx …
The cmt broadcast command
Uniquely sort the dependency graph (leaves first, root last)
Make it an ordered list of packages reached from the current package
Traverse the ordered list
Apply an action to every node
A normal shell command
May stop at first error or continue
May select a subset of the graph
May exclude a subset of the graph
The cmt filter command transforms any text files
expand all $(mmm) and ${mmm} patterns in the file using symbol values
The common options
-quiet
-tag_add=t1,t2
-public
-private
> cmt broadcast gmake
> cmt broadcast –select=/Graphics/ gmake test
Identify and describe the activities
of the software production
Versioning activities
Generally handled through tools like CVS or SVN
CMT provides a plugin to CVS to perform queries about CMT structures
helper command to perform conformant checkout operations: cmt co
SVN is more flexible and can be taught to recognize CMT packages
Through SVN properties
Build activities
Standard compiling and linking is automatically handled from the definitions of constituents
Specialized sequences of operations defined in CMT patterns
Testing, QA assessment
Driven by external tools like JUnit, CppUnit, Qmtest, OVAL, … CodeWizard, RuleChecker
Enforced by automatic applications through CMT patterns
Deployment
CMT query mechanisms to the knowledge base used to generate the manifest files of different packaging systems: Pacman, RPM, Fink
Specify adaptation scheme to configuration parameter values when installed on remote sites

2. Configuration patterns Container and release packages:
CMT
Christian Arnault
The configuration tags
Describe the contextual and operational conditions of the software configuration
The current sub-project
The platform and operating system
The site
The compiler
The step in the life cycle
The user defined conditions
Examples:
Debug or optimized mode
Selection of a graphical environment
Running some special firmware
The version tags of CMT itself
Various production modes:
Some are automatically constructed by CMT
By discovery of the context (OS, compiler version)
To reflect the project strategies
During the activities
While building a constituent
While running a user defined activity (through actions)
Some are externally specified by the user through environment variables
CMTSITE: the current site name
CMTCONFIG: the binary tag name
Tags can be specified on the cmt command line
Using the –tag_add=<tag-list> common option
Tags can be explicitly activated in requirements file
Using the apply_tag <tag> statement
Some are deduced from other tags
Using the tag association statement
Active tags are visualized using cmt show tags
tag i686-rh73-gcc Linux redhat73 gcc-3.2 opt
tag i686-rh73-gcc32-opt Linux redhat73 gcc-3.2 opt
tag i686-rh73-gcc32-dbg Linux redhat73 gcc-3.2 debug
tag i686-rh73-gcc32-pro Linux redhat73 gcc-3.2 opt profiled
tag redhat73&gcc redhat73-gcc32
tag OSF1 Unix
tag Linux Unix
tag HP-UX Unix
tag Darwin Unix
The dependency graph
Describe the primary relationship between packages
Three identification elements
The name
The version request
The directory offset in the project
The dependences form a Direct Acyclic Graph
Clients packages use used packages
Packages may be reached by several paths
CMT supports cycles in the graph
Cycles introduce non deterministic traversal paths
A package exports its public features to its clients
All requirements files of all (visibly) used packages are concatenated to form the complete configuration
Constituents are always private
Automatic (global) patterns are applied to all visible packages
Use statements installed in private sections define a hidden region of the dependency graph when observed from the client perspective
The symbolic parameters
Generically define a configuration parameter
May receive different values under different conditions
According to tag expressions
Is exported to client packages (if installed in a public section)
Can be modified/overridden by client packages
Modifications installed in a private section are not propagated to clients.
Several possible semantics share the same syntax
macro generic internal CMT variable
and make’s macro definition
set environment variable definition
alias alias definition of the shell
path PATH-like environment variable
action shell command
And associated modifiers
<kind>_append
<kind>_prepend
<kind>_remove
<kind>_remove_all
<kind>_remove_regexp
<kind>_remove_all_regexp
macro m “default value” \
Unix “on any Unix” \
rh “only on this machine” \
Win32&debug “for Windows and debug”
macro m “je réutilise $(m1)” \
Linux “but on any Linux”
path_remove PATH “/A/bin”
path_prepend PATH “” \
CERN “/afs/cern.ch/sw/lcg/A/bin” \
LAL “/lal/A/bin”
action directory “ls $(dir_options)” \
Win “dir $(dir_options)”
cmt do directory
Dependency graph
Configuration tags
Constituents
Symbolic parameters
Configuration patterns
Package semantics
Actions
Scoping sections
The constituents
Describe constituents to be constructed from sources
Applications executable
Libraries static, shared, dynamically loadable
Documents anything else
Applications and libraries obey a predefined strategy
Sources are always explicitly specified
Default source location is ../src
But can be explicitly specified
Absolute location
Or relatively to ../src/
Sources are transformed by a language compiler
Support for new languages is described by users
The language statement
Automatic detection of languages from file suffixes
Documents construction method can be freely designed
Make fragments
package doc_provider
make_fragment doc_header
make_fragment doc_trailer
make_fragment doc \
–header=doc_header \
–trailer=doc_trailer
../cmt/fragments/
/doc_header
/doc
/doc_trailer
Provider package
use
doc_header
doc (a.txt)
doc (b.txt)
doc (c.txt)
doc_trailer
package doc_client
document doc mydoc a.txt b.txt c.txt
Client package
mydoc.make
pattern ld_library_path \
path_remove LD_LIBRARY_PATH "" Unix "/<package>/" ; \
path_append LD_LIBRARY_PATH "" Unix "${<PACKAGE>ROOT}/${CMTCONFIG}" ; \
path_remove PATH "" Win32 "/<package>/" ; \
path_append PATH "" Win32 "${<PACKAGE>ROOT}/${CMTCONFIG}"
Configuration patterns
Describe recurrent configuration operations or policies in a project or a software domain
Enforce project policies
Automate and parameterize complex operations
Extend the CMT concepts
Pattern application can be:
Explicit (manual) using the apply_pattern keyword
Application is on a package basis
Automatic when patterns have the –global keyword
Applied to all packages that see the pattern in the dependency graph
Definition of a pattern
A sequence of templated cmt statements
Templates:
Automatic: <package> <PACKAGE> <version>
<path> <project>
User defined: <xxx>
Examples:
Defining the project-wide policy for include search path
Defining automatic shared library access path
Enforcing a systematic QA test application
pattern -global include_path include_path ${<package>_root}/<package>/
pattern quality_test \
application <package>test –s=../test <package>test.cxx <other_sources>
package Pa
use ProjectPolicy
quality_test other_sources=*.cxx
In ProjectPolicy
In package Pa
Container and release packages:
Purely managerial packages
Not involved when using the software
Only provide a CMT requirements file with a set of explicit use statements
No wild cards
Purpose:
Specifies the set of exact version for a release of a software domain
Drive the cmt broadcast commands over the complete software domain
One top container can provide a release package <project>Release.
It describes the complete software release of the project
Every new release of the project implies a new release of the <project>Release container.
Interface packages:
Provide an interface between a software project managed by CMT and a so-called external product (i.e. not managed by CMT)
Specify the native version value defined by the product (as opposed to the CMT version tag)
Adapt the policies and conventions to the project’s ones
Eg Include directory
Construct the compiler and linker options for this product
Set the runtime options (PATH, LD_LIBRARY_PATH, etc…)
Specify the material to export when constructing a distribution kit
Examples
MySQL
ROOT
SEAL
Geant4
Java kits
The LCGCMT project is entirely made of such interface packages
Connect to LCG internal (SEAL, POOL, PI, …) or external products
package XercesC
use LCG_Settings v*
macro XercesC_native_version "2.3.0"
macro XercesC_home "$(LCG_external)/XercesC/$(XercesC_native_version)/..."
include_dirs $(XercesC_home)/include
macro XercesC_linkopts "-L$(XercesC_home)/lib -lxerces-c -lpthread" \
WIN "/LIBPATH:$(XercesC_home)/lib xerces-c_2.lib"
path_prepend LD_LIBRARY_PATH "$(XercesC_home)/lib" WIN32 ""
path_prepend PATH "" WIN32 "$(XercesC_home)/bin"
macro XercesC_export_paths " $(XercesC_home)/include $(XercesC_home)/lib"
macro m1 “a public value”
private
macro m2 “a private value”
use my_package
end_private
# back to public section
Scoping sections
A private section is not exported to the clients
A public specification found in the graph overrides a private region of the dependency graph
The cmt broadcast and cmt show uses reach the private sections by default
The –public common option always hides the private regions
The –private common option always reaches the private regions
Scoping sections end:
At the end of the requirements file
With the end_private or end_public statements
Ending statements return to the previous scope
Policy packages:
Provide policy statements for a software domain
include dir strategy
CMT strategies
CMT patterns
Generic symbols (macros, sets, …)
Examples:
AtlasPolicy is the main policy package, mandatory for all Atlas packages.
ExternalPolicy, AtlasCxxPolicy, LCG_Policy