Advanced Topics in Concurrency and Reactive Programming: Build Tools Majeed Kassis

What are build tools? Definition: Main use cases: build tools are used to compile and construct some form of usable software image from source code. Main use cases: (in) Production Deployment: to compile the finished product, and ship it for sale (in) Continuous Integration: merging all developer working copies to a shared mainline several times a day, after applying a test-suite on the code for validation. (in) Developer Environments: where you are interactively working and changing a codebase, several times a second, and want it to execute many small actions (whether manually or via unit tests) in between modifying the code.

Purpose of build tools Make the development process visible or transparent Provide an easy way to see the health and status of a project Decreasing training time for new developers Bringing together the tools required in a uniform way Preventing inconsistent setups Providing a standard development infrastructure across projects Focus energy on writing applications

Build tools timeline

Evolution of Build Tools

Main features of build tools Compile the code Its main purpose! Supports using external commands to compile code, manipulate images, compress files, etc. Supports downloading dependencies alleviates the need for manual download of dependencies. Supports invoking test suits on code Done prior to compilation to ensure code integrity Supports caching and parallelism Caching output if its inputs did not change in prior compilation iteration Parallelism to speed up the compilation process which can take days

Make

Ant: Another neat tool Ant is a Java based tool for automating the build process Similar to make but implemented using Java Platform independent commands (works on Windows, Mac & Unix) XML based format Avoids the tab issue in make files Ant is an open source Apache project

Project Organization The following example assumes that your workspace will be organized like so… build.xml Project Directory src *.java bin *.class doc *.html

Anatomy of a build File Ant’s build files are written in XML Convention is to call file build.xml Each build file contains A project At least 1 target Targets are composed of some number of tasks Build files may also contain properties Like macros in a make file Comments are within <!-- --> blocks

Projects The project tag is used to define the project you wish to work with Projects tags typically contain 3 attributes name – a logical name for the project default – the default target to execute basedir – the base directory for which all operations are done relative to Additionally, a description for the project can be specified from within the project tag

Build File <project name="Sample Project" default="compile" basedir="."> <description> A sample build file for this project </description> </project>

Properties Build files may contain constants (known as properties) to assign a value to a variable which can then be used throughout the project Makes maintaining large build files more manageable Projects can have a set of properties Property tags consist of a name/value pair Analogous to macros from make

Build File with Properties <project name="Sample Project" default="compile" basedir="."> <description> A sample build file for this project </description> <!-- global properties for this build file --> <property name="source.dir" location="src"/> <property name="build.dir" location="bin"/> <property name="doc.dir" location="doc"/> </project>

Targets The target tag has the following required attribute name – the logical name for a target Targets may also have optional attributes such as depends – a list of other target names for which this task is dependent upon, the specified task(s) get executed first description – a description of what a target does Like make files, targets in Ant can depend on some number of other targets For example, we might have a target to create a jarfile, which first depends upon another target to compile the code A build file may additionally specify a default target

Build File with Targets <project name="Sample Project" default="compile" basedir="."> ... <!-- set up some directories used by this project --> <target name="init" description="setup project directories"> </target> <!-- Compile the java code in src dir into build dir --> <target name="compile" depends="init" description="compile java sources"> <!-- Generate javadocs for current project into docs dir --> <target name="doc" depends="init" description="generate documentation"> <!-- Delete the build & doc directories and Emacs backup (*~) files --> <target name="clean" description="tidy up the workspace"> </project>

Tasks A task represents an action that needs execution Tasks have a variable number of attributes which are task dependent There are a number of build-in tasks, most of which are things which you would typically do as part of a build process Create a directory Compile java source code Run the javadoc tool over some files Create a jar file from a set of files Remove files/directories For a full list see: http://ant.apache.org/manual/coretasklist.html

Initialization Target & Tasks Our initialization target creates the build and documentation directories The mkdir task creates a directory <project name="Sample Project" default="compile" basedir="."> ... <!-- set up some directories used by this project --> <target name="init" description="setup project directories"> <mkdir dir="${build.dir}"/> <mkdir dir="${doc.dir}"/> </target> </project>

Compilation Target & Tasks Our compilation target will compile all java files in the source directory The javac task compiles sources into classes Note the dependence on the init task <project name="Sample Project" default="compile" basedir="."> ... <!-- Compile the java code in ${src.dir} into ${build.dir} --> <target name="compile" depends="init" description="compile java sources"> <javac srcdir="${source.dir}" destdir="${build.dir}"/> </target> </project>

Javadoc Target & Tasks Our documentation target will create the HTML documentation The javadoc task generates HTML documentation for all sources <project name="Sample Project" default="compile" basedir="."> ... <!-- Generate javadocs for current project into ${doc.dir} --> <target name="doc" depends="init" description="generate documentation"> <javadoc sourcepath="${source.dir}" destdir="${doc.dir}"/> </target> </project>

Cleanup Target & Tasks We can also use ant to tidy up our workspace The delete task removes files/directories from the file system <project name="Sample Project" default="compile" basedir="."> ... <!-- Delete the build & doc directories and Emacs backup (*~) files --> <target name="clean" description="tidy up the workspace"> <delete dir="${build.dir}"/> <delete dir="${doc.dir}"/> <delete> <fileset defaultexcludes="no" dir="${source.dir}" includes="**/*~"/> </delete> </target> </project>

Complete Ant Example

Running Ant – Command Line Simply cd into the directory with the build.xml file and type ant to run the project default target Or, type ant followed by the name of a target

Ant Issues Too flexible No structure It can be a bad thing in big projects No structure Each project may have its own structure, forcing developers to learn and understand each project, which wastes valuable time. Too much code to implement simple things: XML XML files become unmanageable even for small projects. Developers need to focus on important tasks Building scripts is not one of them! No dependency management system!

Maven A Java project management and integration build tool. Rather than describing a build as a sequence of commands to run in response to each named target Maven describes the build as a set of modules. Each module lists out metadata such as: Coordinates (name, groupId , and version) Dependencies Characteristics: Supports remote repositories, automatically downloads missing dependencies Maven has a central repository: mvnrepository.com Uses XML format for its configuration file based on Project Object Model. Enforces directory structure for a Maven project Contains dependency management system.

Maven Standard Directory Organization Having a common directory layout would allow for users familiar with one Maven project to immediately feel at home in another Maven project. src/main/java Application/Library sources src/main/resources Application/Library resources src/main/filters Resource filter files src/main/assembly Assembly descriptors src/main/config Configuration files src/main/webapp Web application sources src/test/java Test sources src/test/resources Test resources src/test/filters Test resource filter files src/site Site LICENSE.txt Project's license README.txt Project's readme

Maven: POM.XML Project Management Project Build POM file is the Project Object Model file contains metadata about the project. written in XML syntax. Project Management Detailed description of the project. Company information. Developer roles and information. Mailing list and source control modules configuration. Project Build Source code and test code location. Resources location Project Dependency Libraries needed for build and runtime. Project Reports Junit reports Javadoc reports

Maven Glossary Project: Project Object Model (POM): Artifact: Maven thinks in terms of projects. Everything that you will build are projects. Those projects follow a well defined “Project Object Model”. Projects can depend on other projects, in which case the latter are called “dependencies”. A project may consistent of several subprojects, however these subprojects are still treated equally as projects. Project Object Model (POM): The Project Object Model, almost always referred as the POM for brevity, is the metadata that Maven needs to work with your project. Its name is “project.xml” and it is located in the root directory of each project. Artifact: An artifact is something that is either produced or used by a project. Examples of artifacts produced by Maven for a project include: JARs, source and binary distributions, WARs. Each artifact is uniquely identified by a group id and an artifact ID which is unique within a group.

Maven Glossary GroupId: Dependency: Plug-in: A group ID is a universally unique identifier for a project. While this is often just the project name (eg. commons-collections), it is helpful to use a fully- qualified package name to distinguish it from other projects with a similar name (eg. org.apache.maven). Dependency: A typical Java project relies on libraries to build and/or run. Those are called “dependencies” inside Maven. Those dependencies are usually other projects’ JAR artifacts, but are referenced by the POM that describes them. Plug-in: Maven is organized in plugins. Every piece of functionality in Maven is provided by a plugin. Two types of plug-ins: build plugins, reporting plug-ins Examples of plugins are: jar, clean, javadoc . http://maven.apache.org/plugins/index.html

Project Management Section <pomVersion>3</pomVersion> <groupId>sampleservice</groupId> <name>Sample Service</name> <!-- Used in Javadoc --> <id>sampleservice</id> <currentVersion>1.0</currentVersion> <!-- Used for document creation --> <organization> <name>My, Inc.</name> <url>http://www.myinc.com</url> <logo>/images/logo.gif</logo> </organization> … There are only three required elements in POM. (elements in bold are required)

Project Management Section […] <inceptionYear>2013</inceptionYear> <!-- Used in JavaDoc – <package>com.myinc.sampleservice</package> <!-- Used in JavaDoc --> <shortDescription>Demo to use maven</shortDescription> <!-- one liner --> <!-- Used in front page--> <description> A detailed description about this demo </description> <url>http://www.myinc.com/sampleservice/</url> <issueTrackingUrl/> <siteAddress>dev.myinc.com</siteAddress> <!-- Used in deployment --> <siteDirectory>/www/sampleservice/</siteDirectory> <!-- Used in deployment --> <!-- Used in deployment. If defined, it overrides ${maven.repo.central} --> <distributionSite>/www/dist/sampleservice</distributionSite> <!-- Used in deployment, final distribution directory --> <distributionDirectory>/www/www.myinc.com/somedir</distributionDirectory>

Project Management Section […] <repository> <connection>scm:cvs:pserver:anoncvs@cvs.myinc.com:/cvsroot:sampleservice</connection> <developerConnection>scm:cvs:pserver:${maven.username}@cvs.myinc.com:/cvsroot:sampleservice</developerConnection> <url>http://cvs.myinc.org/viewcvs/sampleservice/</url> </repository> <!-- Used in maven:dist --> <versions> <version> <id>1.0-beta-1</id> <name>1.0-beta-1</name> <tag>1.0-beta-1</tag> </version> </versions> <branches/> … <mailingLists/> <contributors/> <developers/>

Project Dependency Section […] <dependencies> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.8</version> <properties> <ear.bundle>true</ear.bundle> <ejb.manifest.classpath>true</ejb.manifest.classpath> </properties> </dependency> </dependencies> Special Dependency: SNAPSHOT

Project Dependency Section Dependency Classloader […] <dependency> <groupId>bcel</groupId> <artifactId>bcel</artifactId> <version>5.1</version> <properties> <classloader>root</classloader> </properties> </dependency> Maven has three classloaders: root -- ant classloader root.maven – maven core classloader default – plugin classloader

Project Dependency Section Dependency Override project.xml … <dependency> <groupId>weblogic</groupId> <artifactId>weblogic</artifactId> <version>8.1.1</version> <properties> <classloader>root</classloader> </properties> </dependency> project.properties

Project Build Section Defines the location of source, test and resource files. […] <build> <nagEmailAddress>buildmaster@myinc.com</nagEmailAddress> <sourceDirectory>${src.java.dir}</sourceDirectory> <unitTestSourceDirectory>${src.test.dir}</unitTestSourceDirectory> <aspectSourceDirectory/> src/java src/test src/aspect

Project Build Section src/conf <unitTest> <includes> <include>**/*Test.java</include> </includes> <resources/> </unitTest> <resources> <resource> <directory>${src.conf.dir}</directory> <targetPath/> <include>**/*.properties</include> </resource> </resources> src/conf prefix package name, e.g. com.myinc. sampleservice

Project Report Section Defines various reports to be generated <reports> <report>maven-jdepend-plugin</report> <report>maven-checkstyle-plugin</report> <report>maven-changelog-plugin</report> <report>maven-developer-activity-plugin</report> <report>maven-file-activity-plugin</report> <report>maven-javadoc-plugin</report> <report>maven-jxr-plugin</report> <report>maven-junit-report-plugin</report> <report>maven-linkcheck-plugin</report> <report>maven-tasklist-plugin</report> </reports>

Maven Issues Too rigid Using XML format Encoring structure can be a bad thing Due to its enforced structure it can complicate the work on projects which require different structures forcing the developers to work-around it. Using XML format Still requires too much code to implement simple things Polyglot for Maven is a set of extensions that allows non-XML syntax to be used for Maven. However the requirement for back-ward compatibility hinders its quality https://github.com/takari/polyglot-maven

<?xml version="1.0"?> <project name="simple" default="dist" basedir="."> <property name="src" location="src/main/java"/> <property name="srcTest" location="src/test/java"/> <property name="build" location="build"/> <property name="dist" location="${build}/lib"/> <property name="version" value="1.0-SNAPSHOT" /> <path id="classpath.compile"> <pathelement location="libs/commons-lang-2.5.jar"/> </path> <path id="classpath.test"> <pathelement location="libs/junit-4.8.2.jar"/> <pathelement location="${srcTest}"/> <pathelement location="${build}/classes"/> <pathelement location="${build}/test-classes"/> <target name="init"> <mkdir dir="${build}/classes"/> <mkdir dir="${build}/test-classes"/> </target> <target name="compile" depends="init"> <javac srcdir="${src}" destdir="${build}/classes"> <classpath refid="classpath.compile"/> </javac> <target name="testCompile" depends="compile"> <javac srcdir="${srcTest}" destdir="${build}/test-classes"> <classpath refid="classpath.test"/> <target name="test" depends="testCompile"> <junit fork="yes" haltonfailure="yes"> <batchtest fork="yes"> <fileset dir="${srcTest}"> <include name="**/*Test.java"/> </fileset> </batchtest> <formatter type="plain"/> </junit> <target name="dist" depends="test"> <mkdir dir="${dist}"/> <jar jarfile="${dist}/coc-comparison-${version}.jar" basedir="${build}/classes"/> <target name="clean"> <delete dir="${build}"/> </project>

<?xml version="1.0"?> <project name="simple" default="dist" basedir="."> <property name="src" location="src/main/java"/> <property name="srcTest" location="src/test/java"/> <property name="build" location="build"/> <property name="dist" location="${build}/lib"/> <property name="version" value="1.0-SNAPSHOT" /> <path id="classpath.compile"> <pathelement location="libs/commons-lang-2.5.jar"/> </path> <path id="classpath.test"> <pathelement location="libs/junit-4.8.2.jar"/> <pathelement location="${srcTest}"/> <pathelement location="${build}/classes"/> <pathelement location="${build}/test-classes"/> <target name="init"> <mkdir dir="${build}/classes"/> <mkdir dir="${build}/test-classes"/> </target> <target name="compile" depends="init"> <javac srcdir="${src}" destdir="${build}/classes"> <classpath refid="classpath.compile"/> </javac> <target name="testCompile" depends="compile"> <javac srcdir="${srcTest}" destdir="${build}/test-classes"> <classpath refid="classpath.test"/> <target name="test" depends="testCompile"> <junit fork="yes" haltonfailure="yes"> <batchtest fork="yes"> <fileset dir="${srcTest}"> <include name="**/*Test.java"/> </fileset> </batchtest> <formatter type="plain"/> </junit> <target name="dist" depends="test"> <mkdir dir="${dist}"/> <jar jarfile="${dist}/coc-comparison-${version}.jar" basedir="${build}/classes"/> <target name="clean"> <delete dir="${build}"/> </project> <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>grId</groupId> <artifactId>coc-comparison</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <dependencies> <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang</artifactId> <version>2.5</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.1</version> <scope>test</scope> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build> </project>

Polyglot <?xml version="1.0"?> <project name="simple" default="dist" basedir="."> <property name="src" location="src/main/java"/> <property name="srcTest" location="src/test/java"/> <property name="build" location="build"/> <property name="dist" location="${build}/lib"/> <property name="version" value="1.0-SNAPSHOT" /> <path id="classpath.compile"> <pathelement location="libs/commons-lang-2.5.jar"/> </path> <path id="classpath.test"> <pathelement location="libs/junit-4.8.2.jar"/> <pathelement location="${srcTest}"/> <pathelement location="${build}/classes"/> <pathelement location="${build}/test-classes"/> <target name="init"> <mkdir dir="${build}/classes"/> <mkdir dir="${build}/test-classes"/> </target> <target name="compile" depends="init"> <javac srcdir="${src}" destdir="${build}/classes"> <classpath refid="classpath.compile"/> </javac> <target name="testCompile" depends="compile"> <javac srcdir="${srcTest}" destdir="${build}/test-classes"> <classpath refid="classpath.test"/> <target name="test" depends="testCompile"> <junit fork="yes" haltonfailure="yes"> <batchtest fork="yes"> <fileset dir="${srcTest}"> <include name="**/*Test.java"/> </fileset> </batchtest> <formatter type="plain"/> </junit> <target name="dist" depends="test"> <mkdir dir="${dist}"/> <jar jarfile="${dist}/coc-comparison-${version}.jar" basedir="${build}/classes"/> <target name="clean"> <delete dir="${build}"/> </project> <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>grId</groupId> <artifactId>coc-comparison</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <dependencies> <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang</artifactId> <version>2.5</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.1</version> <scope>test</scope> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build> </project> project { modelVersion '4.0.0' artifactId 'coc-comparison' groupId 'grId' version '1.0-SNAPSHOT' dependencies { dependency('commons-lang:commons-lang:2.5') dependency('junit:junit:4.8.2') } build { plugins { plugin { groupId 'org.apache.maven.plugins' artifactId 'maven-compiler-plugin' configuration { source '1.5' target '1.5' Polyglot

Polyglot <?xml version="1.0"?> <project name="simple" default="dist" basedir="."> <property name="src" location="src/main/java"/> <property name="srcTest" location="src/test/java"/> <property name="build" location="build"/> <property name="dist" location="${build}/lib"/> <property name="version" value="1.0-SNAPSHOT" /> <path id="classpath.compile"> <pathelement location="libs/commons-lang-2.5.jar"/> </path> <path id="classpath.test"> <pathelement location="libs/junit-4.8.2.jar"/> <pathelement location="${srcTest}"/> <pathelement location="${build}/classes"/> <pathelement location="${build}/test-classes"/> <target name="init"> <mkdir dir="${build}/classes"/> <mkdir dir="${build}/test-classes"/> </target> <target name="compile" depends="init"> <javac srcdir="${src}" destdir="${build}/classes"> <classpath refid="classpath.compile"/> </javac> <target name="testCompile" depends="compile"> <javac srcdir="${srcTest}" destdir="${build}/test-classes"> <classpath refid="classpath.test"/> <target name="test" depends="testCompile"> <junit fork="yes" haltonfailure="yes"> <batchtest fork="yes"> <fileset dir="${srcTest}"> <include name="**/*Test.java"/> </fileset> </batchtest> <formatter type="plain"/> </junit> <target name="dist" depends="test"> <mkdir dir="${dist}"/> <jar jarfile="${dist}/coc-comparison-${version}.jar" basedir="${build}/classes"/> <target name="clean"> <delete dir="${build}"/> </project> <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>grId</groupId> <artifactId>coc-comparison</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <dependencies> <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang</artifactId> <version>2.5</version> </dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.1</version> <scope>test</scope> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build> </project> project { modelVersion '4.0.0' artifactId 'coc-comparison' groupId 'grId' version '1.0-SNAPSHOT' dependencies { dependency('commons-lang:commons-lang:2.5') dependency('junit:junit:4.8.2') } build { plugins { plugin { groupId 'org.apache.maven.plugins' artifactId 'maven-compiler-plugin' configuration { source '1.5' target '1.5' apply plugin: 'java' version="1.0-SNAPSHOT" group="grId" archivesBaseName="coc-comparison" repositories { mavenCentral() } dependencies { compile 'commons-lang:commons-lang:2.5' testCompile 'junit:junit:4.8.1' Polyglot

Gradle Combines the good parts of both Ant and Maven, while using DSL as a scripting language Allows Ant’s flexibility While keeping Maven’s features. DSL – Domain Specific Language Definition: Small languages presented to solve very specific problem. Example: SQL Glossary: Project: a Jar or an application, each project consists of several tasks. Task: Represent an atomic job to be executed

Gradle Task Task name: hello Executing the task: gradle –q hello Gradle build file: gradle.build

Gradle: doLast doFirst doFirst(Closure action) This method will add your closure as an action to the beginning of the action sequence. All the code that is written in this closure is executed first when the task is executed. doLast(Closure action) This method will add your closure as an action to the end of the action sequence. All the code in this closure is executed in last when the task is executed.

Default Tasks Default tasks will be executed in order if no main task is specified when executing gradle project. In this example: ‘clean’

Task Dependencies Syntax: task.dependsOn task1, task2,…, taskN Task task1(dependsOn: task2, …, taskN) { }

Gradle Repositories

External Dependency Configuration Used to define external dependencies the project requires. Dependency Types: Compile − The dependencies required to compile the production source of the project. Runtime − The dependencies required by the production classes at runtime. By default, also includes the compile time dependencies. Test Compile − The dependencies required to compile the test source of the project. By default, it includes compiled production classes and the compile time dependencies. Test Runtime − The dependencies required to run the tests. By default, it includes runtime and test compile dependencies.

External Dependency Configuration

Gradle Java Plugin Java plugin adds the following capabilities to a project: compilation testing bundling Syntax: apply plugin: <plugin-name> Expected Project Structure: Put all your Java files (full package,etc) into the src/main/java folder Put all your Test files (JUnit stuff) into the src/main/test folder https://docs.gradle.org/current/userguide/java_plugin.html

Java Plugin Build Tasks These default tasks are provided when the java plugin is imported clean will delete a build directory, all your classes will be removed for a fresh compile. assemble will create an archive (JAR) file from your Java project and will run all other tasks that require compile, test and so on.