Karabo installation and software organization Karabo Workshop Hamburg, 24.09.2013 Janusz Szuba WP76 How to edit the title slide Upper area: Title of your talk, max. 2 rows of the defined size (55 pt) Lower area (subtitle): Conference/meeting/workshop, location, date, your name and affiliation, max. 4 rows of the defined size (32 pt)

Outline Overview of development environment Structure and organization of Karabo software Step by step installation

Big picture Package repository Source code maintenance Continuous Integration Code development Documentation Bug tracker

Big picture Package repository Source code maintenance Continuous Integration Code development Documentation Bug tracker

Code development Supported operating systems: Ubuntu 10.04, 12.04 – 32 and 64 bit architecture Scientific Linux 6.3, 6.4 – 32 and 64 bit architecture Mac OS X, 10.7 (Lion), 10.8 (Mountain Lion) – 64 bit architecture Languages C++ (gcc 4.6, 4.7, 4.2 – native Mac OS X compiler) Python 2.7 Build system Make, GNU autotools (for dependencies) NetBeans projects

SVN repository Central SVN repository https://svnsrv.desy.de Authentication – kerberos , basic and certificate Preferred kerberos for all having DESY accounts For external collaborators – basic authentication Web interface https://svnsrv.desy.de/k5websvn/ https://svnsrv.desy.de/k5websvn/wsvn/EuXFEL.WP76/karabo/ Command line access With kerberos authentication https://svnsrv.desy.de/desy/EuXFEL/WP76/karabo With basic authentication https://svnsrv.desy.de/basic/EuXFEL/WP76/karabo

Special arrangement for tutorial SVN repository Special Karabo Tutorial repository Web interface: https://svnsrv.desy.de/k5websvn/wsvn/General.karaboTutorial (only for selected users) https://svnsrv.desy.de/baswebsvn/wsvn/General.karaboTutorial Command line: https://svnsrv.desy.de/desy/karaboTutorial (only for selected users) https://svnsrv.desy.de/basic/karaboTutorial User name and password karabo.workshop@onet.eu, 24Sep2013 When you plan to develop karabo devices or contribute to framework after the workshop Create your own account in SVN repository if you don’t have one already Instructions – Getting Started on Karabo framework site in Alfresco Documentation server Contact us to get access to karabo repository Url to karabo package development https://svnsrv.desy.de/basic/karaboTutorial/karabo/karaboPackageDevelopment

SVN repository structure Karabo repository structure karaboFramework Base libraries, binaries, python bindings, GUI karaboPackages Plugins – devices, sorted in different categories like controlDevices, computeDevices, etc. Additional dependencies for plugins karaboPackageDevelopment Environment for developing and testing plugins Helper script for creating new plugins, commiting, checking out from repository etc. Basic templates with netbeans projects for devices or dependencies Running environment – scripts for starting device servers, gui, broker etc

Code maintenance conventions Follow best practices for a project structure: Trunk, branches, tags folders for each individual project/plugin/dependency Assured when using „karabo” helper script Follow best practices for release versioning: Major, minor, patch level: e.g. 1.3.5 Major number change – possible compatibility break in API, communication protocols Minor numer change – assured API compatibility, new features Patch level change – bug fixes Naming conventions for device plugins, dependencies Camel case for package name, like simpleMotor, simulatedCamera Capitalization for class names, like TestMessageReader Camel case for executables, like tcpClientAsync Coding conventions, netbeans customization

Continuous Integration - Jenkins Automated or on demand tasks: Compiling on different OS Testing – unit tests Creating installation packages Deploying and running functional tests Current setup with Jenkins Nightly builds of karaboFramework from trunk Keeping trunk in shape all the time Release builds karaboFramework installation packages creation, testing and copying to web/ftp repository Jobs on demands, parametrized with svn version, unit tests options, build type, etc Plugin for triggering jobs from netbeans

Installation packages repository Currently contains karaboFramework binary installation packages Binary packages for plugins and dependencies, as well as packaging into rpms and debian packages will follow in future Repository location Internal network http://www-exfel-wp76.desy.de/karabo Accesible from outside ftp://karabo:framework@ftp.desy.de Repository structure Nightly builds: date subfolder, filename indicates svn revision number Released builds: trunk(branch)/version/date, filename indicates version number Links for karabo framework installation packages ftp://karabo:framework@ftp.desy.de/released-builds/tags/1.0.1/2013-09-23 http://exflserv01/karabo/released-builds/tags/1.0.1/2013-09-23

Bug tracker Mantis bug tracker http://www-exfel-wp76.desy.de/static/mantisbt Currently available within internal network Planned migration to a web service available world-wide Register first, then create a new bug report Don’t submit a bug anonymously

Documentation Karabo Framework site on Alfresco documentation server https://docs.xfel.eu/share/page/site/KaraboFramework/dashboard All EuXFEL emploees have account by default For external collaborators, tutorial participants, accounts are created and invitations to join the site sent Wiki pages, Document library Plugin/devices documentation site – Karabo Plugins Accesible from within GUI as a documentation panel Editable in GUI and on the web

Karabo support email karabo@xfel.eu Request Tracker email distribution list Serves as a support entry point

Karabo software use cases Karabo Framework developer Needs all the source code Development environment fully equipped with all necessary dependencies (mainly development packages – header files) Karabo Device Plugin developer Needs Karabo Framework binary package Binaries, libraries, header files, python source Needs plugins source code templates, examples Needs environment for testing developed plugins Karabo software deployment Needs precompiled set of plugins and their dependencies Needs configuration/versioning tool for selecting compatible set of karabo framework and plugins versions For the first two use cases we provide consistent environment setup Deployment use case is in preparation

Karabo Framework software structure Separate netbeans projects Karabo library – base library Code structured to functional groups util, net, io, core, xms, log, xip, webAuth Karathon library – python bindings Binaries Device server, broker message logger Python karabo Natively implemented device server and FSM Python GUI and command line interface Third-party dependencies – compiled and shipped with installation package Python 2.7 with addtional modules: numpy, scipy, matplotlib, ipython (with qtconsole and notebook), pyqt4, h5py Mac OS X needs its own python installation (preferred way through macports) Boost, hdf5, log4cpp, cppunit openMQ C API and JMS broker (currently broker on Linux only)

Karabo Framework binary installation package Precompiled Karabo Framework is available as a self extracting tar file Always installs under single folder karabo-version or karabo- revision_number in case of nightly builds Interactive or batch (needs --prefix=top_installation folder) installation During post-installation phase two configuration files are created $HOME/.karabo/karaboFramework Contains full path to karabo framework installation directory Used in netbeans makefiles for finding location of libraries and header files Every new installation of karabo framework overwrites this file with a new location $HOME/.karabo/karaboSvnPath Contains url pointing to karabo SVN repository for svn command line client Needed as different authentication schemes (kerberos, basic, certificate) have different urls

Karabo Package Development environment Helps with creation, development and running/testing device plugins Set of templates for new packages karabo script new - Create a new package checkout - Checkout an existing package (use list to retrieve name) install - Install a packages configuration update - Update (svn) an installed packages configuration rebuild - Rebuild packages within installed configuration list - Create listings of various type tag - Creates a svn tag for a given package branch - Creates a svn branch for a given package setsvn - Sets karabo svn repository path Run folder Set of scripts for starting up all components: device servers, GUI, CLI, local broker Packages configuration files – describing a logical set of packages to be installed

Karabo Packages Every device plugin is a separate netbeans project In SVN repository it is placed in appropriate category controlDevices, computeDevices, testDevices, etc Can be checkout separately self-contained Netbeans project Can be checkout within Karabo Package Development Helps in development and managing many plugins and their possible additional dependencies Helps with setting-up self-contained running environment

JMS broker Currently using Oracle JMS openMQ 4.5.2 broker Standalone application Or within GlassFish server Also available within full Netbeans installation including GlassFish server For our development team we have centrally maintained broker instance running all the time Communication through different topics – default is a username For standalone development Included in Karabo Framework binary installation package Start/stop/query scripts in Karabo Package Development

Netbeans development environment All karabo software ingredients comes as a separate netbeans project C++ as well as Python Note: there is no official Netbeans support for Python, user community plugin needs to be installed, description on Karabo Framework Wiki Support for command line compilation automated tasks, batch compilation Currently we require Netbeans 7.3 Version 7.3.1 comes with upgraded JMS broker v5.0, not compatible with v4.5.2, currently used with karabo software This is important when running broker from Netbeans

Step by step installation Live demo