1. CSS Build and Update System
Kunal Shroff
EPICS Fall Collaboration Meeting, June 2010
Aix-en-Provence

2. CSS Design Slide stolen from DESY IDE with CSS applications CSS UI
CSS Headless
CSS appl.
CSS appl.
CSS appl.
CSS appl.
UI CSS appl.
CSS appl.
Eclipse IDE
CSS Core UI
CSS Core
Eclipse RCP UI
Eclipse RCP
Eclipse Runtime
OSGi (Equinox)
Java Virtual Machine
Slide stolen from DESY

3. CSS Repository : Architecture
Restructured based on
Functionality and usage
Build system requirements
Criteria
core/: provide contributions to the CSS framework
applications/: plugins/features used by one or more CSS applications
build/: contains the build scripts
product/<siteName>/: site’s product plugin, site specific configuration files and features

4. CSS Repository : Mercurial
Software
CVS
SVN
Git
Mercurial
Atomic commits No
Yes
File rename
Partial
Merge file renames
Symbolic links
Pre/Post event hooks
Signed revisions
Merge tracking
Eol conversion
Tags
Internationsl support
Unicode filename format
Distributed SCM
Improved scalability
Changeset support
Offline operation
no single point of failure
HgEclipse – mercurial plugin for eclipse
DVCS with best support
Clone, push, pull, commit
Add, remove, merge
Browse history
Hudson plugin
Perfect for a collabarative project like css
This is basically true of all the distributed SCMs, but in my experience even more so with Git. There is very little outside of 'fetch', 'pull' and 'push' that communicates in any way with anything other than your hard disk.
This not only makes most operations much faster than you may be used to, but it also allows you to work on stuff offline. That may not sound like a big deal, but I'm always amazed at how often I actually do work offline. Being able to branch, merge, commit and browse history of your project while on the plane or train is very productive
One of the coolest features of any of the Distributed SCMs, Git included, is that it's distributed. This means that instead of doing a "checkout" of the current tip of the source code, you do a "clone" of the entire repository.
This means that even if you're using a centralized workflow, every user has what is essentially a full backup of the main server, each of which could be pushed up to replace the main server in the event of a crash or corruption. There is basically no single point of failure with Git unless there is only a single point.
This does not slow things down much, either. On average, an SVN checkout is only marginally faster than any of the DSCMs. Of the DSCMs I tested, Git was the fastest.
They conceptually organize revisions as changesets which allow you to very easily branch/merge your code.

5. CSS Repository : CVS  Hg
Lossless transition from CVS to Hg
Perfect for a collabarative project like css
This is basically true of all the distributed SCMs, but in my experience even more so with Git. There is very little outside of 'fetch', 'pull' and 'push' that communicates in any way with anything other than your hard disk.
This not only makes most operations much faster than you may be used to, but it also allows you to work on stuff offline. That may not sound like a big deal, but I'm always amazed at how often I actually do work offline. Being able to branch, merge, commit and browse history of your project while on the plane or train is very productive
One of the coolest features of any of the Distributed SCMs, Git included, is that it's distributed. This means that instead of doing a "checkout" of the current tip of the source code, you do a "clone" of the entire repository.
This means that even if you're using a centralized workflow, every user has what is essentially a full backup of the main server, each of which could be pushed up to replace the main server in the event of a crash or corruption. There is basically no single point of failure with Git unless there is only a single point.
This does not slow things down much, either. On average, an SVN checkout is only marginally faster than any of the DSCMs. Of the DSCMs I tested, Git was the fastest.
They conceptually organize revisions as changesets which allow you to very easily branch/merge your code.

6. CSS build The Plug-in Development Environment (PDE)
The usual way of building is using the Export wizard which requires
launching eclipse IDE
Creating the target
Setting up the workspace
Lots of user interaction
Headless build system
Run from command line
Export wizard ??????? build done magically
while this is very useful to get started with it
Need for headless build
Automation
Continous integration
Release process

7. CSS headless build : requirements
Generic Headless build
Simple and easily adaptable for new CSS products
Reuses default build scripts provided by PDE build system
Requires minimum configuration. Minor modification to
Build.properties
Plugin/feature.lists
PDE Build comes with infrastructure to build a complete RCP application from a product configuration file. Most of the setup necessary for such a build can be done through a few modifications to the template build.properties provided in PDE build.
./build.sh nsls2
creates the target against which css is build.
creates the build directory from the details avaliable in the product/site directory
Each sites /css/product/site/
build.properties
plugins.lists and features.lists
category.xml
Now create new file called build.xml (the main build file). This build file was created by Patrick Paulin fromwww.rcpquickstart.com and I have slightly modified it. So you do - the script must prepare the build directory structure (plugins and features directories) and copy the source codes into it. Then it executes the internal Eclipse Ant to start the build process. Beware of wrapped lines!
PDE/Build is a Apache Ant based build system for Eclipse RCP application and Eclipse plugins. This build system is also called "headless" build because it does not require a running Eclipse UI to create your product.
The PDE build system is part of the Eclipse plugin org.eclipse.pde.build.
Build.properties file
Each site has its own
Each site can specify its own update location
Diagram : BuildDirectory Structure.
The first step in setting up a build is to create the directory in which the build will take place. This directory will be referred to as the build directory and will contain the plug-ins and features to build as well as all the generated artifacts. Next, copy your plug-ins and features respectively into "plugins" and "features" subdirectories. Your product configuration file is expected to be in one of these plugins/features.
The second step in setting up a build is to copy the template build.properties file from org.eclipse.pde.build/templates/headless-build to a new directory which will be the build configuration directory (also referred to as the configuration directory). Edit the copied build.properties file and set the following properties: product: the location of your product configuration file in the form "/<plug-in or feature id>/path/to/.product"
baseLocation:  the location of an eclipse install containing all the pre-built features and plug-ins that your product requires in features/ and plugins/ subdirectories.  The RCP delta pack (it is available from the eclipse download page) is mandatory as it includes the org.eclipse.platform.launchers feature which contains the launchers and root files necessary for a product.
buildDirectory:  the directory the build will take place in.  Set this to the full path of the build directory created previously.
configs: list the configurations for which you want your product to be built. You can uncomment the configuration(s) provided (be careful of the line continuations).
archivePrefix: the name of the directory of your product once installed on disk.

8. CSS headless build : build steps
./build.sh NSLS2
Create Target (the group of plugins the CSS product is built against)
Downloads and untars eclipse RCP + Delta pack + all other third party plugins.
Create BuildDirectory (all the plugins/features to be built)
Collects the core,application and site specific plugins and features to be built
Build (uses the PDE build system)
Launches “headless” eclipse.
Builds using antRunner applications and the default product.xml build scripts.
PDE Build comes with infrastructure to build a complete RCP application from a product configuration file. Most of the setup necessary for such a build can be done through a few modifications to the template build.properties provided in PDE build.
./build.sh nsls2
creates the target against which css is build.
creates the build directory from the details avaliable in the product/site directory
Each sites /css/product/site/
build.properties
category.xml
Now create new file called build.xml (the main build file). This build file was created by Patrick Paulin fromwww.rcpquickstart.com and I have slightly modified it. So you do - the script must prepare the build directory structure (plugins and features directories) and copy the source codes into it. Then it executes the internal Eclipse Ant to start the build process. Beware of wrapped lines!
PDE/Build is a Apache Ant based build system for Eclipse RCP application and Eclipse plugins. This build system is also called "headless" build because it does not require a running Eclipse UI to create your product.
The PDE build system is part of the Eclipse plugin org.eclipse.pde.build.
Build.properties file
Each site has its own
Each site can specify its own update location

9. CSS Continuous Integration System Using Hudson
Building/testing software projects continuously
Detects and fixes integration problems continuously
Automate Deployment – publishes the update site to sourceforge
Makes the build process transparent.
Saves time and effort
Continuous Integration Key Practices: The continuous integration key practices are the required steps to avoid the anti-patterns identified above. Skipping these will eventually start causing the team to fall into the anti-patterns which starts to affect the overall quality, reliablility, and speed in which the developers can deliver the code. Following these practices is something that the team as a whole has to enforce. If the development team does not make a clean build a priority, then why are you continuously building? As specified by the article, the key practices are: Maintain a Single Source Repository - this means that their should be one master source repository that is built from. If you are using GIT or a distributed version control system, there still should be a master repository that all changes are eventually migrated too.
Automate the Build - builds should be automated. They need to run with out human intervention. The should also be able to run on any system with out modification. Anybody should be able to run a build locally on their own machine. No machine specific dependencies.
Make Your Builds Self-Testing - unit tests, functional tests, and acceptance tests need to be incorporated into the build. Including build reports.
So far the EPDP which many eclipse projects follows, seems to be doing alright in regards to the key practices. However, at this point it devates and starts following the anti-patterns outlined above. Adding the following key practices can help eliminate them. Everyone Committs Every Day - it's important for every developer on the team to synch with the main repository at least once a day. Ideally more frequently. According to Martin Fowler, "By doing this frequently, developers quickly find out if there's a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it's easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve."
Every Committ Should Build the Mainline on an Integration Machine - MAP file only builds are not sufficient for continuous integration. They are a valid overall build step, but code should also be built from the head as often as possible.
Keep the Builds Fast - A common reason that is used for not building from Head, is that the builds take too long. The longer the build takes the less feedback to the developers and the long potential integration problems go un-addressed. Extreme Programming recommends a build should take no longer than 10 minutes. Martin Fowler suggests a Staged Build, "The idea behind a staged build (also known as build pipeline) is that there are in fact multiple builds done in sequence. The commit to the mainline triggers the first build - what I call the commit build. The commit build is the build that's needed when someone commits to the mainline. The commit build is the one that has to be done quickly, as a result it will take a number of shortcuts that will reduce the ability to detect bugs. The trick is to balance the needs of bug finding and speed so that a good commit build is stable enough for other people to work on." Maybe not every unit test needs to be run. Maybe building the particular components and bring in dependencies is enough. Other longer running builds can be run, i.e. MAP file, or the current integration builds. The point is to provide feedback early and often to catch problems while they are easy to find and fix.
Test in a Clone of the Production Environment - Ideally, you have a build that runs on all of the platforms that you will eventually deploy too. If you are deploying to Windows, Linux, and Mac. You should have builds that run on all of those environments.
Make it Easy for Anyone to Get the Latest Executable - This is where the Eclipse Development Process causes an anti-pattern. Section 6.4 is the culprit. It specifically makes it difficult to find the latest builds. It says they should not be visible to the the general public. However this makes it much more difficult for adopters and potential early users to help find bugs. Nightly, and I-Builds should be made available to everyone.
Everyone can See what is Happening - Builds Status should be viewable by anyone. Continuous Integration is all about communication. Communication to the developers. Communication to the adopters. Communication to the users.
Automate Deployment - Deployement to the end users machine should be as simple as possible. If you are building a product to be distrubuted, then the installation process should be simple. This may mean creating the necessary installer scripts.

10. CSS Common Product Continuous Integration with Hudson
Ensures satisfaction of Build requirements
This can be done by simply adding a dependency to the XYZ plugin in the product definition file.
The plugin also needs to be added to the plugins.list in the cs-studio/product/COMMON/plugins.list.
Make sure that all the plugins required by XYZ are also added to both the product definition and the plugin.lists file.
Does not guarantee that nothings broken
org.cssstudio.testsuite
Css-nsls2 p2 enabled
// contains the required p2 plugins ( currently packaged in the p2.ui feature) this is similar to the feature that was used in the eclipsecon p2 tutorial
// some pesky update manager plugins(3) are still present – need to check the plugins form core that depend on them
The update site or p2 repository is built during the p2 integrated headless build.
These updates can be consumed by the css applications in various ways.
Each site can distribute their version of css with a set of predefined update sites

11. Eclipse p2 – update management
p2 provides a provisioning platform for Eclipse-based applications
Update your RCP and OSGi applications
Provide a mechanism to install and manage new features
Provide a mechanism to manage different configurations
Manage your developers target platforms
p2 can provision
OSGiTM bundles
Executables
Configuration files – for instance eclipse.ini and config.ini
Registry keys
Native code
Also supports bundle pooling – sharing bundles across multiple installs
P2 manages all the dependencies – if it installs then you know all the dependencies have been satisfied
Installation and updates are managed in Eclipse using a provisioning platform called p2. Fundamentally, p2 is a technology for provisioning and managing Eclipse- and Equinox-based applications. You can use p2 to install or manage any aspect of your application, from the physical plug-ins and native code, to the configuration of the installed software - file permissions, command line arguments, etc. Installation with p2 does not consist simply in adding or removing files in the file system, but more generally the sequence of events that must occur to lay down and configure a system that is ready to run. Core aspects of p2 include:
Automatic resolution of dependencies between software components. With p2, you state the root set of items you want installed or uninstalled, and p2 computes the complete set of required system changes automatically.
Transport of software components and configuration data from remote repositories to the system being installed. p2 includes sophisticated algorithms for performing multi-threaded transfers, including support for mirrors and automatic re-balancing of transfers based on network state (availability, throughput).
An extensible mechanism for "teaching" p2 how to install and configure various kinds of software. By default p2 knows how to install and configure Eclipse plug-ins, and basic native integration such as adding/removing files, setting permissions, and creating symbolic links.
A graphical user interface integrated into the Eclipse platform, to allow end users to examine and manage the application.
A suite of command line tools and Ant tasks, to allow developers and release engineers to build and configure p2-enabled applications.

12. CSS-NSLSII p2 enabled product CSS-NSLSII update site
Install/update preferences.
List of update sites known to CSS
Update configuration options
Self updating applications
Update from cloud
User initiated update
Css-nsls2 p2 enabled
// contains the required p2 plugins ( currently packaged in the p2.ui feature) this is similar to the feature that was used in the eclipsecon p2 tutorial
// some pesky update manager plugins(3) are still present – need to check the plugins form core that depend on them
The update site or p2 repository is built during the p2 integrated headless build.
These updates can be consumed by the css applications in various ways.
Each site can distribute their version of css with a set of predefined update sites

13. CSS-NSLSII update
All the screen shots of the entire update process

14. Conclusions Restructured repository Generic headless build system
Continuous Integration with hudson
p2 for installation and update management

15. Questions