SKA/KAT SPIN Presentation Software Engineering (!?) Robert Crida

Overview Software Configuration Management (SCM) Automated Build System External Dependencies Design Study – Antenna Control Software Module (ACSM)

SCM Challenge KAT is expected to have a life span in excess of 20 years (project will outlive the people) Maintenance is inevitable during that period Continuous development is also likely Importance over that period of being able to freeze build environment What about hardware failure that far into the future?

SCM Use subversion – potentially multiple repositories (collaborations, different subsections) Store multiple projects in a tree Standard svn hierarchy for each project  trunk (primary development)  tags (releases)  branches (qa, users, etc) Try to include all dependencies as projects within the hierarchy

Subversion Layout Tools (used by the build process)  doxygen tags  doxygen  doxygen  etc (typically only store tags for each tool) Code (ultimately linked into products)  ThirdPartyLibraries (typically only tags)  InternalProjectsHierarchy Each layer can only depend on other projects in the same or lower layers, ideally wanted to only depend on projects in lower levels but deemed too difficult for users, instead have to work out build order within each layer.

Build system based on distutils  Actually we use from recursivebuild import setup  recursivebuild.setup delegates to setuptools.setup and provides a convenient place to centrally add new commands (eg doc, pylint)  Use setuptools because of egg support Allows multiple builds of the same library to be present simultaneously  from pkg_resources import require  require(“numpy==1.0”)  import numpy Most distutils projects easily converted by changing import of setup function Python projects

Build System Each directory of the hierarchy has a script to build its children  script determines order  passes arguments from parent  arguments modeled after distutils.setup Leaves are the projects themselves, must provide script understanding build targets  Python projects use setup.py  Other projects provide eg build.py to wrap make or scons etc

Build system contd 1 All scripts must understand standard build targets  install (more like build) puts python eggs into suitable directory should compile and link other libs but not deploy  test (runs unit tests on the project)  doc (runs doxygen on project)  pylint (runs pylint on project – rename to lint!)

Build system contd 2 Build controller machine stores svn repository mirror plus overall build script Build environment to be configured on a virtual machine, currently requires only gcc and python Virtual machine is cloned prior to each build svn repository mirror is updated and copied to the build vm Multiple build steps are executed remotely on the vm by the controller

Build system contd 3 This allows multiple vm's with different compiler versions and even OS versions Each can be built using (probably) the same control script

Testing Integrate unit testing based on xUnit into the auto- build Investigating Fitnesse based testing for automated ATP type testing Looked at web testing technologies such as Twill and selenium to assist with automated web app testing

Dependencies Python projects can depend on other python projects using the egg mechanism, hence can just find suitable egg in standard PYTHONPATH Depend on other projects by specifying paths within the hierarchy  multiple versions of a lib can coexist  requires extensive include and lib paths  use $PROJECT_ROOT environment variable instead of relative paths due to svn moves (different nesting levels)

External Dependencies Try to minimize external dependencies  all need to go into SCM  increases code-base (potential maintenance problem)  traded off against leveraging existing code AIPS++  many dependencies  hard to build – difficult to configure tools  constantly changing Use each one's own build system  must configure appropriate include and lib paths

External Python Dependencies Eggs help a lot – in theory Despite distutils, many python packages have non- standard builds which makes life difficult Haven't yet found a python project which defaults to using setuptools