1. Integrating Instrument SW Packages
Description:
This presentation explains the concepts of INS SW Package and Integration Module, to then show how an actual instrument SW package is built.
Concepts and tools to support packages were developed on the basis of a great deal of common characteristics found in the installation procedures of VLT instrumentation software.
The goal was to achieve a standard for INS SW installation, that every application must use.
P. Baksai

2. Purpose
Show you how to get INS SW built, installed, configured, and ready to run
An example of an actual INS SW Package configuration

3. Instrument SW Package Contains: INS SW Modules of specific versions
CCS Environment templates
Instrument configuration
Tools
It is created by means of an Integration Module
A special SW module
One for operations site
Another for each development and test site
Lists the other modules to get from software repository
A software package contains one special software module called the ‘integration module’.
This software module is the first module retrieved from the CMM archive and is placed in the
package root directory. This module usually contains all the code or configuration files needed
to retrieve the rest of the software modules of the package, and to build and install all modules,
Environments, etc.

4. Integration Module Defines an INS SW Package Package name Sub-packages
Locality pre-requisites
Targets
Module versions, compilation sequence and options
Common installation procedures
Developer defined installation steps (plugins)
CCS environment configuration
User environment setup
For an instrument named e.g. XXXX, the software prefix is usually xx, and the application will run in a wxxxx workstation.
The operations user would be xxxx and the software manager user xxxxmgr.
Integration module is named xxins and must contain at least config/xxinsINSTALL.cfg
These are VLT standards.
Then:
Package name: XXXX
Sub-packages: Related software modules inside the package are stored in subdirectories (or sub-packages).
E.g. the software package that contains the XXXX instrumentation software may have the following structure:
XXXX
|--DCS
|--xxd
|--...
|--ICS
|--xxi
|--OS
|--xxo
|--MS
|--xxmcfg
|--xxins
Locality pre-requisites: a set of environment variables and associated ranges that must be defined before the installation can begin
Targets: the operational site has the full set of hardware, other sites might have a limited set. Each site is referred to with a target name
Module compilation sequence: one module could require other modules compiled before for using their libraries and/or includes
Common installation procedures: all instrument SW installation share steps such as create root directories, retrieve modules from CMM archive, compile, create CCS environments.
Developer defined installation steps: for a particular instrument SW, customized steps can be intercalated in the series of standard steps.
CCS environment configuration: env process table, database loader files
User environment setup: operational user environment facilities such as menus, workspaces, colors, audible warnings.

5. Tools pkgin: building and installing packages on targets
pecs: locality management
cmm: source code repository
stoo: application startup tool
ctoo: configuration tool
We are going to focus on pkgin for the rest of the presentation

6. pkgin It is part of the VLT Common SW
Current version in VLT2008: pkgin 1.112
INPUTS
Reads PAF configuration files from the integration module that defines the package
OUTPUTS
Creates INTROOT, INS_ROOT and the environments
INPUTS: At least a default configuration file, e.g. xxinsINSTALL.cfg, must exist.
OUTPUTS: The INTROOT and INS_ROOT are created if they do not exist, if they do, they are updated.

7. pkgin Architecture Drawing borrowed from VLT-MAN-ESO-17240-1913
Dashed line represent optional items

8. Example
Building
Configuration

9. Building an INS SW Package (1)
First time on target
Check TARGET env. var.
Check INTROOT, INS_ROOT parents exist
Check the environments are registered in ACC
Run vccmake as vltmgr on the target machine
Test xxxxmgr account can access CMM
Test rsh access from xxxxmgr to xxxx accounts
Test rsh access from xxxx to TCS workstation
In a standard VLT environment, suppose you have your integration module xxins already checked out and the xxinsINSTALL.cfg file ready.
When the package is being built on the target for the first time, some extra checks and tests will save some time and effort:
TARGET: This env. var. must have the value PARANAL for the operational target. For development targets it must be registered in the integration module
INTROOT and INS_ROOT: They are created if do not exist, but their parent directories must exist. /vlt/XXXX for INTROOT and /data/XXXX for INS_ROOT are the standard parent directories
vccmake: this script creates hosts and services tables and other OS resources needed to create CCS environments
CMM access: The manager account needs to retrieve source code from CMM, for that to work, a password for xxxxmgr must exist in ~xxxxmgr/.cmmrc
rsh access: The manager account will startup environments remotely as xxxx user, xxxx user will run some tools remotely in TCS workstation

10. Building an INS SW Package (2)
Rebuilding on target
Locate your integration module or check it out from source repository
Then, just type:
wxxxx > pkginBuild xxins
Wait…, check the logs, startup the applications
wxxxx> xxinsStartup
The idea is that pkgin takes care of everything for a safe rebuild: save config, stop, backup, restore
Environments are always backed up, if you want more safety use keywords INSTALL.CON.BACKUP “T” and INSTALL.CON.MAXBAK “1” to backup INTROOT and INS_ROOT also.

11. Operations Target PAF File (1)
Featured examples taken from foins for FORS2
PAF file with full configuration for operations target
PAF.HDR.START; # Start of PAF Header
PAF.TYPE "Configuration"; # Type of PAF
PAF.ID "FORS2 Control SW"; # ID for PAF
PAF.NAME "foinsINSTALL.cfg"; # Name of PAF
PAF.DESC "FORS2 Package"; # Short description of PAF
PAF.CRTE.NAME "Pedro Baksai"; # Name of creator
PAF.CRTE.DAYTIM "2003/11/01"; # Civil Time for creation
PAF.LCHG.NAME " "; # Name of person/appl. changing
PAF.LCHG.DAYTIM " "; # Timestamp of last change
PAF.CHCK.NAME " "; # Name of appl. checking
PAF.HDR.END; # End of PAF Header
INSTALL.PACKAGE.ENVVARS "INSTRUMENT FORS_ID TCSID";
INSTALL.PACKAGE.NAME "FORS2";
INSTALL.PACKAGE.TARGETS "PARANAL";
In the example taken from FORS2, the package requires some environment variables to be defined; it will install only for PARANAL target

12. Operations Target PAF File (2)
Append other PAF files
# VLTSW
INSTALL.APPEND1.FILE "foinsINSTALL_VLTSW.$(VLTSW_RELEASE).cfg";
# INS COMMON
INSTALL.APPEND2.FILE "foinsINSTALL_COMMON.cfg";
# FORS SHARED
INSTALL.APPEND3.FILE "foinsINSTALL_FORS.cfg";
# FORSi SPECIFIC CONFIG.
INSTALL.APPEND4.FILE "foinsINSTALL_$(INSTRUMENT).cfg"
The INSTALL.APPENDi.FILE keyword allows to modularize configuration

13. Operations Target PAF File (3)
Type and templates for the environments #
# ENVIRONMENTS #
INSTALL.RTAPENV1.NAME "wf$(FORS_ID)ors"
INSTALL.RTAPENV1.TPLSRC "foins/ENVIRONMENTS/wf$(FORS_ID)ors"
INSTALL.LCUENV1.NAME "lf$(FORS_ID)ics1"
INSTALL.LCUENV1.TPLSRC "foins/ENVIRONMENTS/lf$(FORS_ID)ics1"
INSTALL.LCUENV1.BOOTENV "$INSTALL.RTAPENV1.NAME";
INSTALL.LCUENV2.NAME "lf$(FORS_ID)ics2"
INSTALL.LCUENV2.TPLSRC "foins/ENVIRONMENTS/lf$(FORS_ID)ics2"
INSTALL.LCUENV2.BOOTENV "$INSTALL.RTAPENV1.NAME";
INSTALL.LCUENV3.NAME "lf$(FORS_ID)ics3"
INSTALL.LCUENV3.TPLSRC "foins/ENVIRONMENTS/lf$(FORS_ID)ics3"
INSTALL.LCUENV3.BOOTENV "$INSTALL.RTAPENV1.NAME";
Name and template for environments according to their type: RTAP or LCU

14. Operations Target PAF File (4)
Detector name, WS and DB file #
# DETECTORS #
INSTALL.DET1.NAME "FIERA";
INSTALL.DET1.TYPE "FIERA";
INSTALL.DET1.CCDNAME "ccdF";
INSTALL.DET1.CCDLENV "wf$(FORS_ID)ccd";
INSTALL.DET1.DBFILE "fcdSciFors.dbcfg";
The INSTALL.DET1.DBFILE is needed for the INSTALL_CCD step

15. Operations Target PAF File (5)
Scan link to LCU, detector, TCS #
# SCANLINKS #
INSTALL.SCAN1.REMOTEENV "$INSTALL.LCUENV1.NAME";
INSTALL.SCAN1.LOCALENV "$INSTALL.RTAPENV1.NAME";
INSTALL.SCAN1.FILE "ficsScnData.lf$(FORS_ID)ics1";
INSTALL.SCAN7.REMOTEENV "$INSTALL.DET1.CCDLENV";
INSTALL.SCAN7.LOCALENV "$INSTALL.RTAPENV1.NAME";
INSTALL.SCAN7.SCRIPT "DCS";
INSTALL.SCAN8.REMOTEENV "wt$(TCSID)tcs";
INSTALL.SCAN8.LOCALENV "$INSTALL.RTAPENV1.NAME";
INSTALL.SCAN8.SCRIPT "tifTcsScanSetup -dev -p :Appl_data:TCS -t wt$(TCSID)tcs";
Highlight the reference of previously defined configuration keywords

16. Operations Target PAF File (6)
Source code modules
INSTALL.MODULE.CPU "MC68040"; #
# FORS2 PACKAGE MODULES #
# Reserved indexes
INSTALL.MODULE70.NAME "f2otsf";
INSTALL.MODULE70.SUBPKG "OS";
INSTALL.MODULE70.VERSION "1.43";
INSTALL.MODULE72.NAME "f2oseq";
INSTALL.MODULE72.SUBPKG "OS";
INSTALL.MODULE72.VERSION "0.44";
INSTALL.MODULE49.NAME "exp";
INSTALL.MODULE49.VERSION "1.102";
INSTALL.MODULE49.SUBPKG "OS";
INSTALL.MODULE49.MAKE "OPTIMIZE=2 USER_CFLAGS=-DFMON clean all man install";
INSTALL.MODULE49.OPTIONS "IGNORE_WARNINGS";
Highlight the use of reserved index range, make options, and build options

17. Operations Target PAF File (7)
Files and Hooks #
# FILES
# -----
INSTALL.FILES1.NAME "BOBRC";
INSTALL.FILES1.DESTUSER "$INSTALL.LOGINNAME.USER";
INSTALL.FILES1.LINK "foins-all-FORS$(FORS_ID).bobrc .bobrc";
INSTALL.FILES2.NAME "MENU";
INSTALL.FILES2.SRCHOST "$INSTALL.RTAPENV1.NAME";
INSTALL.FILES2.DESTHOST "wu$(TCSID)uws";
INSTALL.FILES2.DESTUSER "$INSTALL.LOGINNAME.USER";
INSTALL.FILES2.FILE "foins-misc-uws.env .pecs/misc-all.env";
INSTALL.FILES2.FILE "foins-misc-all.xrdb .pecs/misc-all.xrdb";
INSTALL.FILES2.FILE "foins-misc-FORS$(FORS_ID).wmrc .pecs/misc- all.wmrc";
The MENU files are installed in a remote host: the User Workstation

18. Operations Target PAF File (8)#
# HOOKS
# -----
INSTALL.HOOK5.NAME "AFTER_START_WSENV"
INSTALL.HOOK5.PLUGIN "foinsInstallInsRoot"
INSTALL.HOOK6.NAME "AFTER_INSTALL_CCD"
INSTALL.HOOK6.PLUGIN "foinsAfterInstallCcd"
INSTALL.HOOK8.NAME "AFTER_START_WSENV"
INSTALL.HOOK8.PLUGIN "rrmInstall"

19. Test Site Target PAF File (1)
fompar
Can append PAF files from other module (e.g. foins)
PAF.HDR.START; # Start of PAF Header
PAF.TYPE "Configuration"; # Type of PAF
PAF.ID "FORS Control SW"; # ID for PAF
PAF.NAME "fomparINSTALL.cfg"; # Name of PAF
PAF.DESC "FORS Package"; # Short description of PAF
PAF.CRTE.NAME "P. Baksai"; # Name of creator
PAF.CRTE.DAYTIM "2006/06/26"; # Civil Time for creation
PAF.LCHG.NAME " "; # Name of person/appl. changing
PAF.LCHG.DAYTIM " "; # Timestamp of last change
PAF.CHCK.NAME " "; # Name of appl. checking
PAF.HDR.END; # End of PAF Header
INSTALL.PACKAGE.TARGETS "DEV_WS";
INSTALL.APPEND1.MODULE "foins"
INSTALL.APPEND1.FILE "foinsINSTALL.cfg"
Again using FORS2 as an example:
The main PAF file must be called fomparINSTALL.cfg under fompar/config; it can append keywords from foins main PAF file.
This package will install in development targets exporting TARGET=DEV_WS

20. Test Site Target PAF File (2)
Replaces some keywords, adds new ones, according to test needs
# Replace workstation env so that the alarm logger and server is not
# started - the simulated devices on the LCUs generate 1000s
# alarms.
INSTALL.RTAPENV1.TPLSRC "fompar/ENVIRONMENTS/wf$(FORS_ID)ors";
# simulated TCS: wfxtcs.
INSTALL.RTAPENV2.NAME "wf$(FORS_ID)tcs";
INSTALL.RTAPENV2.TPLSRC "fompar/ENVIRONMENTS/wf$(FORS_ID)tcs";
INSTALL.SCAN8.REMOTEENV "$INSTALL.RTAPENV2.NAME";
INSTALL.SCAN8.LOCALENV "$INSTALL.RTAPENV1.NAME";
INSTALL.SCAN8.SCRIPT "tifTcsScanSetup -l wf$(FORS_ID)ors -p :Appl_data:TCS -t $(INSTALL.RTAPENV2.NAME)";
INSTALL.SCAN8.SRCATTR "Appl_data:TCS.version";
INSTALL.SCAN8.DESTATTR "Appl_data:TCS:info.version";
Appending keywords from fomparTARGET_DEV_WS.cfg may overwrite same keywords if found in foinsINSTALL*.cfg
Other PAF file for the DEV_WS target can overwrite instrument configuration for ICS, OS and start, which are installed from f2mcfg module:
fomparINS_DEV_WS.cfg would replace keywords from f2mcfgINS.cfg, for example to put all devices in simulation
fomparOS_DEV_WS.cfg would replace keywords from f2mcfgOS.cfg, for example to define TEST mode, increase timeout
fomparSTART_DEV_WS.cfg would replace keywords from f2mcfgSTART.cfg, for example to increase log level, put LCUs in SIM mode

21. Configuration File good practices
Try to modularize using the INSTALL.APPENDi.FILE keyword
Reserve index ranges of INSTALL.MODULEi.* keywords, for each sub-package
Use environment variables if they are target dependent
If not, try to use a FITS config keyword instead
Reference environment vars only if you have them checked with INSTALL.PACKAGE.ENVVARS
INSTALL.RTAPENV1.NAME "wf$(FORS_ID)ors"
Try to reference FITS keywords instead of writing duplicate values
INSTALL.SCAN1.LOCALENV "$INSTALL.RTAPENV1.NAME"
Modularity is always good ;-)
Reserving indexes prevents nasty problems when duplicated keywords are used by mistake in different appended PAF files
If you need to reference environment vars other than a very standard one like VLTSW_RELEASE, then include them in INSTALL.PACKAGE.ENVVARS list to have them checked at start of pkginBuild.
AS far as possible, reference already defined keywords to simplify maintenance

22. A lot more features, the manual is worth reading
Flexible configuration
You can override tcl code
You could make a library from a module
Separate options for WS code and LCU code
Not enough time here for a complete coverage of the subject

23. Other useful things you can do with pkgin
Get help
wxxxx> pkginBuild -help
Run a sub-sequence of two or more steps
wxxxx > pkgunBuild xxins -fromstep RETRIEVE -tostep BUILD_MODULES
Build a single module
wxxxx > pkginBuild -step BUILD_MOD mymod
Build a single environment
wxxxx > pkginBuild xxins -step BUILD_ENV -env lxxics1
Compare module versions
wxxxx > pkginVersions xxins
Clean module compilations
wxxxx> pkginClean xxins

24. Version Comparison Example
Module Source/INTROOT Last Config -> Update Comments
iser LOCKED: PPRS-17951
bob > UPDATE
volac
cnfg / > UPDATE
calob / RETRIEVE
autlog > UPDATE RETRIEVE
f1mseq / > UPDATE INSTALL
f1ocal / > UPDATE RETRIEVE
foins / LAST LAST RETRIEVE LOCKED
iser: it is locked to version 1.16 for the reasons explained in PPRS-17951
bob: a newer version exists, config should be updated
volac: it is up to date
cnfg: a newer version exists that was already installed manually, configuration must be updated
calob: a newer version exists that was already configured and installed manually, the source code should be retrieved from CMM
autlog: a newer version was configured but not installed nor retrieved
f1mseq: a modified version was installed, configuration must be updated and latest version installed again
f1ocal: a newer than configured version was installed but it is not the latest, updated configuration and retrieve the source code
foins: always retrieve the last version from CMM

25. Package Verify Example
Module iser installed: OK.
Module bob installed: OK.
Module volac installed: OK.
Module cnfg wrong version FAILURE.
Module calob wrong version FAILURE.
Module autlog wrong version FAILURE.
Module f1mseq wrong version FAILURE.
Module f1ocal wrong version FAILURE.
Module foins wrong version FAILURE.
It is not very consistent for autlog, and not very informative in terms of what is wrong and what to do.
That´s why we all prefer to use pkginVersions for the analysis.

26. Conclusion Integration Module helps to:
Implement VLT SW standard environment for instruments
Maintain Configuration Control
Encapsulates knowledge of the INS SW
Deliver INS SW package
Pkgin tool helps to:
Automate installation procedure
Verify system consistency
Easily update the application version
Minimize risk of installation mishaps
Implement the VLTSW Standard Environment
$VLTROOT $INTROOT $INS_ROOT $VLTDATA
|_/bin |_/Sources |_/SYSTEM/DETDATA |_/ENVIRONMENTS
|_/config |_/config |_/SYSTEM/COMMON/CONFIGFILES |_/config
|_/tmp
~xxxxmgr/. ~xxxx/.
|_XXXX_Sources
Support Configuration Management & Control
- Use of CMM archive to retrieve a defined software configuration
Allows new developments while preserving the current official package
Control and traceability of changes
Encapsulates knowledge
You don´t need to remember all commands and the correct sequence
Makes package maintenance independent from specific people
Delivers INS SW package
If you are a developer and are asked to hand over the software package you just say “OK, fetch xxins 1.0”

27. References
VLT-MAN-ESO : Installation Tool For VLT SW Packages, issue 5, 2005/12/31
ftp://ftp.eso.org/pub/vlt/vlt/pub/releases/JAN2006/vol- 5c/VLT-MAN-ESO pdf