1. CSS Build and Update System
Kunal Shroff
EPICS Spring Collaboration Meeting, June 2010
Aix-en-Provence

2. CSS Design Slide stolen from DESY IDE with CSS applications CSS UI
CSS Headless
CSS appl.
CSS appl.
CSS appl.
CSS appl.
UI CSS appl.
CSS appl.
Eclipse IDE
CSS Core UI
CSS Core
Eclipse RCP UI
Eclipse RCP
Eclipse Runtime
OSGi (Equinox)
Java Virtual Machine
Slide stolen from DESY

3. CSS Mercurial repository
Distributed revision control
Offline operation
Changesets based
HgEclipse – mercurial plugin for eclipse
DVCS with best support
Clone, push, pull, commit
Add, remove, merge
Browse history
Perfect for a collabarative project like css
This is basically true of all the distributed SCMs, but in my experience even more so with Git. There is very little outside of 'fetch', 'pull' and 'push' that communicates in any way with anything other than your hard disk.
This not only makes most operations much faster than you may be used to, but it also allows you to work on stuff offline. That may not sound like a big deal, but I'm always amazed at how often I actually do work offline. Being able to branch, merge, commit and browse history of your project while on the plane or train is very productive
One of the coolest features of any of the Distributed SCMs, Git included, is that it's distributed. This means that instead of doing a "checkout" of the current tip of the source code, you do a "clone" of the entire repository.
This means that even if you're using a centralized workflow, every user has what is essentially a full backup of the main server, each of which could be pushed up to replace the main server in the event of a crash or corruption. There is basically no single point of failure with Git unless there is only a single point.
This does not slow things down much, either. On average, an SVN checkout is only marginally faster than any of the DSCMs. Of the DSCMs I tested, Git was the fastest.
They conceptually organize revisions as changesets which allow you to very easily branch/merge your code.

4. CSS build The Plug-in Development Environment (PDE)
The usual way of building is using the Export wizard which requires
launching eclipse IDE
Creating the target
Setting up the workspace
Lots of user interaction
Headless build system
Run from command line
Export wizard ??????? build done magically
while this is very useful to get started with it

5. CSS headless build Generic Headless build ./build Site specific
Download & untars eclipse in a temp directory
Creates the buildDirectory by collecting all the required plugins and features
Site specific
product plugins/feature
build.properties
category.xml
are obtained from the css/product/site/…
PDE Build comes with infrastructure to build a complete RCP application from a product configuration file. Most of the setup necessary for such a build can be done through a few modifications to the template build.properties provided in PDE build.
./build.sh nsls2
creates the target against which css is build.
creates the build directory from the details avaliable in the product/site directory
Each sites /css/product/site/
build.properties
category.xml
Now create new file called build.xml (the main build file). This build file was created by Patrick Paulin fromwww.rcpquickstart.com and I have slightly modified it. So you do - the script must prepare the build directory structure (plugins and features directories) and copy the source codes into it. Then it executes the internal Eclipse Ant to start the build process. Beware of wrapped lines!
PDE/Build is a Apache Ant based build system for Eclipse RCP application and Eclipse plugins. This build system is also called "headless" build because it does not require a running Eclipse UI to create your product.
The PDE build system is part of the Eclipse plugin org.eclipse.pde.build.
Build.properties file
Each site has its own
Each site can specify its own update location

6. CSS Continuous Integration System Using Hudson
Building/testing software projects continuously
Detects and fixes integration problems continuously
Automate Deployment – publishes the update site to sourceforge
Makes the build process transparent.
Saves time and effort
Continuous Integration Key Practices: The continuous integration key practices are the required steps to avoid the anti-patterns identified above. Skipping these will eventually start causing the team to fall into the anti-patterns which starts to affect the overall quality, reliablility, and speed in which the developers can deliver the code. Following these practices is something that the team as a whole has to enforce. If the development team does not make a clean build a priority, then why are you continuously building? As specified by the article, the key practices are: Maintain a Single Source Repository - this means that their should be one master source repository that is built from. If you are using GIT or a distributed version control system, there still should be a master repository that all changes are eventually migrated too.
Automate the Build - builds should be automated. They need to run with out human intervention. The should also be able to run on any system with out modification. Anybody should be able to run a build locally on their own machine. No machine specific dependencies.
Make Your Builds Self-Testing - unit tests, functional tests, and acceptance tests need to be incorporated into the build. Including build reports.
So far the EPDP which many eclipse projects follows, seems to be doing alright in regards to the key practices. However, at this point it devates and starts following the anti-patterns outlined above. Adding the following key practices can help eliminate them. Everyone Committs Every Day - it's important for every developer on the team to synch with the main repository at least once a day. Ideally more frequently. According to Martin Fowler, "By doing this frequently, developers quickly find out if there's a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it's easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve."
Every Committ Should Build the Mainline on an Integration Machine - MAP file only builds are not sufficient for continuous integration. They are a valid overall build step, but code should also be built from the head as often as possible.
Keep the Builds Fast - A common reason that is used for not building from Head, is that the builds take too long. The longer the build takes the less feedback to the developers and the long potential integration problems go un-addressed. Extreme Programming recommends a build should take no longer than 10 minutes. Martin Fowler suggests a Staged Build, "The idea behind a staged build (also known as build pipeline) is that there are in fact multiple builds done in sequence. The commit to the mainline triggers the first build - what I call the commit build. The commit build is the build that's needed when someone commits to the mainline. The commit build is the one that has to be done quickly, as a result it will take a number of shortcuts that will reduce the ability to detect bugs. The trick is to balance the needs of bug finding and speed so that a good commit build is stable enough for other people to work on." Maybe not every unit test needs to be run. Maybe building the particular components and bring in dependencies is enough. Other longer running builds can be run, i.e. MAP file, or the current integration builds. The point is to provide feedback early and often to catch problems while they are easy to find and fix.
Test in a Clone of the Production Environment - Ideally, you have a build that runs on all of the platforms that you will eventually deploy too. If you are deploying to Windows, Linux, and Mac. You should have builds that run on all of those environments.
Make it Easy for Anyone to Get the Latest Executable - This is where the Eclipse Development Process causes an anti-pattern. Section 6.4 is the culprit. It specifically makes it difficult to find the latest builds. It says they should not be visible to the the general public. However this makes it much more difficult for adopters and potential early users to help find bugs. Nightly, and I-Builds should be made available to everyone.
Everyone can See what is Happening - Builds Status should be viewable by anyone. Continuous Integration is all about communication. Communication to the developers. Communication to the adopters. Communication to the users.
Automate Deployment - Deployement to the end users machine should be as simple as possible. If you are building a product to be distrubuted, then the installation process should be simple. This may mean creating the necessary installer scripts.

7. Eclipse p2 – update management
p2 provides a provisioning platform for Eclipse-based applications
Update your RCP and OSGi applications
Provide a mechanism to install and manage new features
Provide a mechanism to manage different configurations
Manage your developers target platforms
p2 can provision
OSGiTM bundles
Executables
Configuration files – for instance eclipse.ini and config.ini
Registry keys
Native code
Also supports bundle pooling – sharing bundles across multiple installs
P2 manages all the dependencies – if it installs then you know all the dependencies have been satisfied
Installation and updates are managed in Eclipse using a provisioning platform called p2. Fundamentally, p2 is a technology for provisioning and managing Eclipse- and Equinox-based applications. You can use p2 to install or manage any aspect of your application, from the physical plug-ins and native code, to the configuration of the installed software - file permissions, command line arguments, etc. Installation with p2 does not consist simply in adding or removing files in the file system, but more generally the sequence of events that must occur to lay down and configure a system that is ready to run. Core aspects of p2 include:
Automatic resolution of dependencies between software components. With p2, you state the root set of items you want installed or uninstalled, and p2 computes the complete set of required system changes automatically.
Transport of software components and configuration data from remote repositories to the system being installed. p2 includes sophisticated algorithms for performing multi-threaded transfers, including support for mirrors and automatic re-balancing of transfers based on network state (availability, throughput).
An extensible mechanism for "teaching" p2 how to install and configure various kinds of software. By default p2 knows how to install and configure Eclipse plug-ins, and basic native integration such as adding/removing files, setting permissions, and creating symbolic links.
A graphical user interface integrated into the Eclipse platform, to allow end users to examine and manage the application.
A suite of command line tools and Ant tasks, to allow developers and release engineers to build and configure p2-enabled applications.

8. CSS-NSLSII p2 enabled product CSS-NSLSII update site
Install/update preferences.
List of update sites known to CSS
Update configuration options
Self updating applications
Update from cloud
User initiated update
Css-nsls2 p2 enabled
// contains the required p2 plugins ( currently packaged in the p2.ui feature) this is similar to the feature that was used in the eclipsecon p2 tutorial
// some pesky update manager plugins(3) are still present – need to check the plugins form core that depend on them
The update site or p2 repository is built during the p2 integrated headless build.
These updates can be consumed by the css applications in various ways.
Each site can distribute their version of css with a set of predefined update sites

9. CSS-NSLSII update
All the screen shots of the entire update process

10. Conclusions Restructured repository Generic headless build system
Continuous Integration with hudson
p2 for installation and update management

11. Questions