Build system And Software Implementation Standard by Andreas Schuh

 Introduction  Project Template  CMake Modules  Utilities  Milestones  Tutorials 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.2

Why BASIS? What is it? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.3

 the lifetime of your software to be longer than the research time spent on the development of the algorithm it implements.  Therefore, we want  to establish a unified software structure.  to ease integration of packages.  to reduce maintenance burden.  to make our software more robust against failure.  to increase the impact factor of your work! 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.4

 Specify standards/conventions for the software development process.  Standardize project organization and software build including testing and packaging.  Provide basic utility functions in order to promote compliance with lab conventions and to reduce common development burden. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.5

9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.6 BASIS …ToolkitODVBA COMPARE PREDICTGLISTRMICO Integrated Distribution Package Projects COMPARE PREDICT … ToolkitBASIS

 BASIS is not a toolkit!  It focuses on more basic aspects.  It consists of specifications in plain text or on Wiki.  It accompanies this standard with a preset project template and basic utility functions for supported programming languages.  However, distinction may be fuzzy at times… 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.7

9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.8 Standard Specifications Template Filesystem Hierarchy Template Files CMake Modules BasisProject.cmake (CMake) BasisTest.cmake (CTest) BasisPack.cmake (CPack) Tools Project Creation Automated Testing Documentation Utilities Command Execution Command-line Parsing Testing Frameworks

How does it look like? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.9

 is the installation prefix, e.g., /usr/local/ on Unix.  is a project- and possibly version-specific string, e.g.,. / bin/ / include/sbia/ / lib/ cmake/ / perl5/5.x.x/ SBIA/ pythonx.x/ site- packages/ sbia/ share/ / doc/ example/ …  is the root directory of the source tree.  is the project name in lowercase only. / config/ data/ doc/ example/ include/sbia/ / src/ test/ 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.10

 Note:  We distinguish between main executables in /bin/ / and auxiliary executables located in /lib/ /.  By default, symbolic links are created upon installation in the /bin/ directory which link to (selected) executables in /bin/ /.  More details in the specification.specification 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.11

 Common project files which are part of the project template and copied upon project creation.  For an example, see root CMakeLists.txt template.CMakeLists.txt 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.12

What are they good for? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.13

 Implement functions and macros which extend and replace most of the standard CMake commands.  Know not only C++, but also other languages: Java (not yet), Python, Perl, BASH, and MATLAB.  Implement filesystem hierarchy standard, and adhere to guidelines regarding software documentation, packaging, and testing.  basis_add_executable() 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.14  add_executable()

 The main modules are:  BasisProject.cmake ▪ To be included by every BASIS project. Initializes BASIS and includes definition of basis_* functions and macros.  BasisTest.cmake – Extends CTest.cmake. ▪ Used for software tests. ▪ Included by basis_project_initialize().  BasisPack.cmake – Extends CPack.cmake. ▪ Used for packaging. ▪ Included by basis_project_finalize(). 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.15

 The most important functions to know are:  basis_find_package()  basis_add_executable()  basis_add_library()  basis_add_test()  basis_add_doc()  basis_include_directories()  basis_target_link_libraries()  basis_set_target_properties()  basis_get_target_property()  basis_install() 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.16

 /config/Depends.cmake # find NIfTI-1 tools for MATLAB basis_find_package (MatlabNiftiTools REQUIRED) basis_add_include_directories (“${MatlabNiftiTools_INCLUDE_DIRS}”)  /src/CMakeLists.txt # MEX-file build from C++ basis_add_library (look_for_gold_instead MEX look_for_gold.cxx) # executable build from MATLAB scripts basis_add_executable (miner neurominer.m) basis_target_link_libraries (miner look_for_gold_instead) 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.17

 See the API documentation of BASIS for details on the CMake variables, functions, and macros.API documentation  The documentation is generated automatically every night (less frequently in later stages) from the in-source code comments using Doxygen. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.18

What tools do we get? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.19

 Used to create a new BASIS project or to modify a previously created project.  Can upgrade a previously created BASIS project to a newer version of BASIS.  Uses 3-way file merge to merge your changes with the changes in the newer BASIS template.  Therefore, stores meta-data in.basis/ directories in project source tree. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.20

 Wrapper for command-line tools implemented for the automated execution of software tests:  The basistest_cron is executed, e.g., every 5 minutes.  It in turn just calls basistest_master with the arguments specific to the given system.  A configuration file tells the basistest_master what branches of which projects to test at which time interval,…  The actual tests are executed by instances of basistest_slave which are submitted to SGE in our case and am using CTest. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.21

 See CDash dashboard of BASIS for examples of test result submissions of bothdashboard  Continuous ▪ Executed after a change was committed to the repository. and  Nightly ▪ Run once every certain time interval, e.g., every night. tests. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.22

 Natively, Doxygen knows how to parse C/C++, Java, and Python (not satisfactory, though)  Filters can be used to translate other languages into something Doxygen understands best: C++.  BASIS includes such filters for the missing languages, i.e.(, Python), Perl, BASH, and MATLAB.  Application of filters hidden to the developer. A simple basis_add_doc(api GENERATOR Doxygen) in the build configuration is sufficient. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.23

 See the API documentation of the BASIS utilities for examples of documentation generated from in-source comments only.API documentation 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.24

What implementations does it provide? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.25

 Executing several utility commands from within one or more main commands is common practice on Linux and in particular in research environment.  Several factors affect which commands are actually executed and which shared libraries are loaded.  How can we make sure that the right commands are executed with the correct libraries loaded? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.26

 BASIS provides utility functions for each supported language to do just that in a safe way.  Further, these functions satisfy the Calling Conventions, where executables are referenced by build target name rather then filename.Calling Conventions 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.27

 Any command-line tool needs to parse the given command-line arguments.  Each main executable is required by BASIS to output a help screen on --help (-h), a shorter usage information on --helpshort, and its version and copyright on --version. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.28

 Therefore, BASIS provides command-line parsing libraries for each supported language.  These libraries create a help screen from brief descriptions provided for each option.  No need to format the help screen manually.  Possibility to output a man page or XML description.  Such help is more likely to be up-to-date rather then being forgotten when an option was renamed, removed, or added… 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.29

 C++  TCLAP TCLAP  gflags gflags  Java  args4j args4j  Python  argh (wrapper for argparse) arghargparse  Perl  Getopt::Tabular Getopt::Tabular  Getopt::Declare Getopt::Declare  BASH  shflags (extended shFlags) shflagsshFlags  MATLAB  none available yet 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.30 See Command-line Parsing.txt document in BASIS repository for more details.Command-line Parsing.txt

 C++  gtest gtest  gmock gmock  libtap++ libtap++  Java  JUnit JUnit  JTap JTap  Python  unittest unittest  PyTap PyTap  Perl  Test::Simple Test::Simple  Test::More Test::More  Test::MockObject Test::MockObject  BASH  shtap (slightly modified tap-functions library) shtaptap-functions  MATLAB  MATLAB xUnit Test Framework MATLAB xUnit Test Framework 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.31 Already part of Perl installation. Test Anything Protocol (TAP) Producers Common use and standardized output which can be processed, for example, by Test::Harness written in Perl. Already part of Python installation.

What’s the plan? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.32

 Release date: Monday, 10/3/2011  Stable features:  Languages: C++, Python, Perl, BASH, MATLAB  Project Template.  CMake Modules.  Tools basisproject and basistest.  Build of executables from MATLAB scripts.  Build of MEX-files from C++.  Command Execution using build target name.  See Wiki for further release details.Wiki 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.33

 Yes, it is beta. Thus,  for ongoing projects, it is up to you whether to migrate to BASIS or postpone this for until prior to publication. ▪ Plan ahead! This might take some efforts. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.34  for new projects, it is strongly recommended to use the beta release of BASIS already. ▪ This makes you familiar with it and reduces the burden of adopting it once version 1.0 is released. ▪ Your feedback and experience in using BASIS is important to shape the stable release version. ▪ Before the stable release, changes can be easily made and decisions can be reconsidered. ▪ After such release, it’s not!

 Release date: Monday, 11/7/2011  Stable features:  Command-line parsing.  Build of shared libraries from MATLAB scripts.  Tutorial-like How-To Guides. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.35

 Release date: Monday, 12/12/2011  New stable features:  Testing frameworks.  Improved Doxygen filters for Python and Perl. 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.36

 Release date: Monday, 2/27/2012  New stable features:  Packaging, including creation of source packages that bundle also external dependencies and considerations regarding decision that were be made by then on how to distribute our software and make it noticed. ▪ licensing permitting  Coding Guidelines  More How-To Guides 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.37

Please help me! 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.38

 Short tutorial-like meetings to go more into details regarding certain aspects with hands-on examples over the next few months.  Invitations will be send in advance via SBIA mailing list including links to tutorial materials.  Remember, your participation and feedback is required to form the first release version of BASIS!  What do you like?  What concerns you?  What hinders you?  What is missing? 9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.39

9/30/2011BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved.40 Let’s talk about it!