1. Overview of the Automated Build & Deployment Process Johnita Beasley Tuesday, April 29, 2008

2. Overview of Deployment Tiers Software builds and deployments performed by the QA Team Software builds and deployments performed by the Systems Team Software builds and deployments performed by the Systems Team Inside the NIH Firewall Outside the NIH Firewall DEV QA STAGE PROD

3. Build/Deployment Process (Current) 1. Generate & Test the Ant build script. Ant Build Script DEV 2. Test the Ant build script and Deployment on a Production-like server Desktop 3. Generate Build & Deployment Instructions for QA Build Request & Deployment Instructions QA 4. Test the Build & Deployment STAGE PROD 5. Tests app. Sends Build & Deployment To Systems Team Build Request & Deployment Instructions 6. Deploys Application Based on Build & Deployment Instructions 6A 6B

4. Guiding Principles (Automated) One process to rule them all. The same build and deployment process will be used on all tiers, without exception. This enables ongoing validation and will reveal problems more quickly. In a broader sense, this principle also means that all teams will use the same approach. Simplicity. Where multiple alternatives existed, the team favored the one that seemed simpler and more direct. Standardization. This involved standardizing on a directory structure, creating a master set of environment properties, and designing a standard build script. This approach benefits the Center as a whole, because developers who work on more than one project, and open source customers who implement more than one of our products locally, only have to learn one way of doing things. Tool-neutrality. Build scripts must be executable from the command line in a properly configured environment. The Center uses AnthillPro as a Build Management Server, but open source users of our software may not have access to it. Therefore, we will not inject any dependencies on AnthillPro into our process.

5. Guiding Principles (Automated) Tier-neutral builds. The output of the build step should be a tier-neutral deployment unit, and a set of external property files, which implies that a build script should not have tier specific build targets. Appropriate error handling. Build scripts should adopt a ‘fail early’ approach to detecting and reporting errors. Database scripts should perform extensive error checking and should be able to rollback to the previous state if any error is encountered during execution. Development teams are responsible for providing for both confirmation of a successful deployment, and rollback of failed deployments.

6. Central Build Server Used as a ‘reference’ build machine Supports isolation of build and deployment configuration parameters Configured with the necessary external dependencies Anthill Pro build management server will fulfill build and deployment requests on all tiers Training will be provided to teams in use of the Anthill Pro tool

7. Hot Deployments NCICB uses shared application containers Applications must be hot-deployable to be considered eligible for deployment at arbitrary time without disrupting operation of other applications Applications that are NOT hot-deployable (require a container restart) will only be deployed during regularly scheduled deployment windows.

8. Master set of tier-specific properties Systems team will provide to all developers a standard set of environment properties and their appropriate values. This file will describe the deployment environment to which their applications must conform. A version of this file will be maintained on each tier and the appropriate version will be supplied to the build process All tier-specific properties will be included in the master file. Ensures all projects will use a consistent naming scheme.

9. Directory Structure Standardization Influenced by the Eclipse web application defaults (since Eclipse is the tool of choice for most development teams) Detailed description provided in the Automated Build and Deployment Handbook

10. Build Script Standardization Two primary build targets recommended are: – build-all: creates the (tier-neutral) deployment unit, and copies all output to the deployment-artifacts directory – deploy: first checks to see that the deployment-artifacts directory exists and the deployment unit is there before proceeding with the deployment These targets are the primary interface points for all users of the script There should be no direct link between these targets

11. Standard for reading build property files A build script using this approach can be invoked in one of two ways: a. >ant build-all – Load build properties from a local file called build.properties b. >ant -DPROP.FILE=/path/to/my.properties build-all – Overloads the default value of PROP.FILE and loads properties from the specified file The benefits of this approach are: – The build script can be run with an arbitrary set of parameters – Allows the Systems Team to maintain protected versions of properties files for STAGE & PROD – Keeps all tier-specific properties in easily accessible place

12. Naming Conventions The file prefix “template.” indicates build and deploy substitution variables are present. The expanded file will have the same file name without the “template” prefix, e.g. template.web.xml will be used to create web.xml after the substitution variables are expanded, using Ant's filtering capabilities. Property names should be all UPPER CASE for external build.xml properties and all lower case for internal build.xml properties. This is to make it easier to determine where a property’s definition can be found. Properties that are directory or path references should include “.DIR” or “.dir” in their names. Properties that refer to files should include “.FILE” or “.file” in their names. Properties that refer to environment variables should use “ENV.” in their names. Target names begin with a dash (‘‘) for internal build.xml targets. In addition to the conventions outlined here, the project team has adopted the recommendations of The Elements of Ant Style, available at http://wiki.apache.org/ant/TheElementsOfAntStyle.http://wiki.apache.org/ant/TheElementsOfAntStyle

13. Property validation Each build script is responsible for validating it’s properties For most cases, it will only be possible to validate that the property actually has a value Validating of meaningful values should be performed if possible – Example: Verifying the correct versions of Java and Ant are being used Both the JAVA_HOME and ANT_HOME environment variables will be available during script execution and should be checked for the appropriate values. If any validation step fails, an appropriate error message should be generated and the script halted immediately.

14. Property Validation Example The 1 st segment of the following code snippet validates all properties, and sets the check.props property if a required property is not set. It is at the top of the script and immediately evaluated before any targets are invoked. In the 2 nd segment, the ‘–prop-check’ target is a dependent of every other target and will cause the script to fail if check.props is set. Failure Message

15. Summary of Build Step Build script reads a tier-specific set of properties passed in on the command line. Build script verifies that all required properties have values. If possible, the script may also check to ensure that the properties have meaningful values. This is the required approach for all teams implementing auto deployment. Build script creates tier-neutral build artifacts, and copies them all to the deployment artifacts subdirectory.

16. Summary of Deploy Step Based on the tier-specific properties selected at deploy time, the script should copy the deployment units to the specified directory on the deployment machine. After the files are successfully copied over, the build script should move them to their final destination on the target machine. This can be accomplished this by using the Ant SSHEXEC task to execute remote commands on the target. The build machine has been configured with SSH keys to enable appropriate access on the target machine.

17. Example: Execution of Remote Commands The example code is reusable as is, without modification, to any project that follows the recommended directory structure. <scp todir="${SCP.USER}@${SCP.HOST}:${SCP.PATH}" port="${SCP.PORT}" passphrase="${SCP.PASSPHRASE}" keyfile="${SCP.KEYFILE}" trust="true" verbose="true">

18. Responsibilities Systems/SCM Team – Maintains a master file of properties (provided to development teams) that describes the build and deployment environment – Provides implementation support for teams automating their build scripts through all tiers – Providing access to the Anthill Build Server Development Teams – Writing and maintaining build scripts – Addressing build failures

19. Related Documents and References Deployment Handbook Automated Build and Deployment Handbook You can get these documents from: http://gforge.nci.nih.gov/docman/index.php?group_id=27&selected_doc_group_id=84&language_id=1

20. Questions