1. A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Software Process & Infrastructure for LCG Project Overview (38 slides, 22 screen dumps) EGEE Meeting 28 October 2003 Alberto AIMAR

2. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 2 Project context of LCG SPI LCG Application Area LCG Infrastructure Common services Similar ways of working (process) Tools, templates, training General QA, tests, integration, release LCG Application Area software projects POOL: Persistency SEAL: Core common software PI: Physics Interfaces SIMU: Simulation …etc… LHC grid software applications (LHC experiments, projects, etc) LCG SPI project

3. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 3 Project context of the LCG SPI “Software Management Process RTAG” General recommendations -All LCG projects must adopt the same set of tools, standards and procedures -Adopt commonly used open-source or commercial software when easily available -Avoid “do it yourself solutions” -Avoid commercial software, if may give licensing problems If each project needs an infrastructure, many projects need it even more… -Tools, standards and procedures -Try to avoid complexity

4. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 4 Infrastructure Software Development a. Provide general services needed by each project -CVS repository, Web Site, Software Library -Mailing Lists, Bug Reports, Collaborative Facilities b. Provide solutions specific to the software phases -Tools, Templates, Training, Examples, etc. Coding Analysis and Design Development Release Specifications Testing ….. Deployment and Installation ….. Planning General Services Software Development

5. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 5 Project guidelines No specific software development The project should avoid any new development, and future maintenance, of any specific software package needed for the LCG software infrastructure. Use HEP or free software All software used should be already available, or become available, in the HEP community or in the free software community The goal of the project is to define an infrastructure and later a simple process Future maintenance will need separate planning and resources. Everything is done in collaboration LCG and LCG projects (Pool, Seal, etc) LHC experiments IT division Big projects (G4, Root, etc) Using as the existing IT services The project should use all existing IT services Make sure to match the LCG needs

6. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 6 SPI project strategy Have different and separated services Simple solutions Work with the users Meeting face to face Establish simple deliverables Develop as little as possible Provide common services We did not start to provide tools for requirements, design, etc.. We started from development-related work (repository, releases, testing, bug report, etc)

7. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 7 SPI work break-down Break the project in “components” -Each is a sub-project -A responsible person in the LCG SPI -Understand and learn the subject -Know/find who knows about the subject -Provide practical solutions, usable independently User Requirements Analysis and Design DevelopmentRelease DistributionTesting Documentation Deployment and Installation External Software ….. Planning

8. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 8 SPI internal organization Each component of the infrastructure has: A responsible person in the project Similar approach -Define the goal of the component -Standard procedures and documentation Standard procedure Survey of possible/existing solutions in HEP and free software Meet the people expert and responsible of the component in real projects (LCG or experiments or big projects) Discuss and agree/decide/verify a solution Present the solution Implement the solutions and make it available Use in the LCG SPI project itself Use it in a real project (LCG or experiment or big project)

9. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 9 LCG decision process The SPI project follows what is decided by the LCG management and by the projects and experiment via the Architects Forum Actually we always involve as much as possible the real users and developers and value all help and feedback But at some point one must decide and we must provide something LCG management support is crucial to SPI especially for what concerns QA, policies issues and standardizations of usage of tools

10. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 10 SPI services The rest of this talk is devoted to describe the SPI services and activities within the SPI project

11. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 11 http://spi.cern.ch

12. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 12

13. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 13 CVS repository and Delivery Areas CVS repository A central CVS repository managed and available to all projects Any project just needs to ask for it, and declare its users permissions Managing mirroring and backups Users access is controlled Tools for automatic clean up of locks, etc Will be moved to use the IT CVS service Delivery areas AFS area an area to install software created by projects in the LCG application area (lcg/apps) an area for external and third party software (lcg/external) an area for software under evaluation within a project (lcg/contrib)

14. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 14

15. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 15  Code Documentation Features of interest Code browsing Code searching Code information Various design/data diagrams Any LCG project will have them available as part of the infrastructure Doxygen  extracts comments, builds documentation and diagrams LXR  connects the source code and allows search in the code ViewCVS  allows browsing of the CVS repository from the web

16. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 16

17. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 17 Code documentation: Doxygen

18. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 18 Code documentation: LXR

19. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 19 Code documentation: ViewCVS CVSgraph: Displays the tree of revisions and branches graphically Enscript: Colorize files in the CVS repository.

20. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 20 Configuration and build system The tools selected by LCG was SCRAM (build system saga) All projects are currently building with Scram SPI is providing a common configuration for all projects In term of which tools and versions to use SCRAM is used in a different way from project to project Transfer of knowledge to LCG people (in all projects) is difficult Supporting the tool is complicated it is used in different ways not very good quality software The improvements needed are not completely there Speed issues, porting to Windows, improving efficiency, separate configure from make Fixes instead of proper solutions and the result is not very satisfactory It was difficult to provide a tutorial and a good support

21. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 21 http://spi.cern.ch/scram

22. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 22 Nightly build system Builds periodically the LCG software Runs the tests Presents the results We did not look for external or other tools but currently Nicos is being developed Provided by BNL/Atlas (A.Undrus) Derived from what is being developed in Atlas The author is very motivated to support it for the LCG Work is still in progress

23. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 23

24. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 24 LCG workbook Provide a central place for the documentation Introduction to new users in the LCG Task-oriented Web-based Inspired by the Babar workbook but we are still far from there Needs contributions from all projects SPI provides the part about the infrastructure

25. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 25 http://spi.cern.ch/workbook

26. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 26 LCG Policies CVS Directory Structure Policy C++ Source Code Guidelines Build Directory Policy Software Testing Policies Version Numbers, Tagging and Release Procedure Installation Directory Structure Platform string, binary names, debug flags and more They are a needed by the whole LCG, not by SPI They are defined by the LCG projects, not by SPI If everything is different is too difficult to use and to automate Is just matter of compromising on our habits, for project needs We do not stress the policies a lot, but we gather them We have to tell when they are not followed This is a job assigned to SPI, first time that this is done so is sometimes difficult to achieve it “easily”

27. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 27

28. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 28 External Software Service (slides by Eric Poinsignon) We install software needed by LCG projects. We provide a set of Open Source and Public Domain software (libraries and tools) like: Compilers (icc, ecc) Scientific libraries (GSL) CERN made packages (clhep, root) General tools (python) Test tools (cppunit, qmtest) Database software (mysql, mysql++) Documentation generators (lxr, doxygen) XML parsers (XercesC) There are currently 45 different software, plus others under evaluation.

29. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 29 External Software organization The LCG projects (SEAL, POOL, PI, Simulation and SPI) propose what to install and in which version When there is agreement with the experiments Here is the list of the available platforms (decided by the Architect Forum) Linux RedHat 7.3 with the compilers -gcc 3.2 (rh73_gcc32) -icc 7.1 (rh73_icc71) -ecc 7.1 (rh73_ecc71) Windows -Visual Studio.NET 7.1: (win32_vc7). SPI installs the software, often lot of help from users 45 software packages on different platforms and several versions totalizing more than 300 different installations

30. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 30 How does External Software work? A unique AFS location: /afs/cern.ch/sw/lcg/external/ (even for software already existing under AFS). Standard installation structure: software_name/version/platform/software_content i.e.: Boost/1.30.2/rh73_gcc32/(lib, include, bin, …) Web information: one page per software. Homepage: http:/spi.cern.ch/extsoft/ The name of the person who helped installing the software is mentioned there.

31. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 31

32. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 32

33. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 33 External Software configuration Following a “How to”, many software are installed by users Each installation log is stored under _SPI directories accessible from the web or AFS. i.e.: Boost/1.30.2/rh73_gcc32/_SPI/install.txt and install_log.txt All the software can be used with configuration management under the SCRAM ToolBox: One description file per software containing library names, include files location and other variables. the chosen versions. the path to each package A single tag (like LCG_20) to get the whole ToolBox configuration. A central solution reduces the work for all the users, avoids duplications and guarantees uniformity.

34. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 34 External Software Service users LCG Application Area projects People in experiments and laboratories The installations are publicly available on AFS For developers that need local installation, A distribution process has been developed to ease the copy of our software in different locations (see J. Moscicki presentation: Software Distribution) It exploits the SCRAM toolbox to extract the needed packages. In terms of shortlist, version and platform.

35. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 35 Savannah Project Portal (slides by Yves Perrin) The Web portal for LCG SW projects Customized from GNU (SourceForge as origin) Based on Apache, PHP, mySQL Functionality: Bug tracking Task management Mailing lists, news, faqs Access to CVS repository Download area, etc Project and user oriented views

36. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 36 LCG Savannah homepage

37. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 37 Project homepage

38. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 38 Overall approach and benefits Common set of services, centrally managed: exposure to a large community common structure, functionality and interface no maintenance & upgrade load on the projects Configurable for the projects needs Configurable for the user needs and automatically adapted to his / her rights

39. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 39 Project’s bug administration

40. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 40 Benefits Totally web based Single entry point to project(s) Uniform access to project information Set up common web infrastructure for a project without coding

41. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 41 Creation of projects (still active today)

42. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 42 Support load (for SPI) What SPI changed installation from GNU, general bug fixing and improvements implemented bulk user registration integration with AFS authentication sending these improvements back to GNU What SPI does administration (project approval) maintenance (submitted bugs) development (support requests) staying in phase with GNU and keep contact with other developers

43. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 43 Status 70 hosted projects, 395 registered users major new release in preparation (merge of CERN and GNU branches, common tracker for all services, etc) we work closely with the open source people still a few minor bugs and limited documentation (online help & faqs)... will be fixed gradually LCG Savannah is accessible at http://savannah.cern.ch

44. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 44 Software testing should be an integral part of the software development in the LCG App Area. All level of software testing should be run as part of an automatic process. SW-Testing Overview (slides by Manuel Gallas Torreira) GOAL (from RTAG2) Software testing Integration test System test Acceptance test Unit test Sw-testing team Integration Tests System Tests Examples LHC experiments Use in Exp. Test Frameworks SW-testing doc SW- testing support SPI SW-Testing SW-testing polices Code Documentation Software developer CODE Work Package Test Automated testing Nightly Building System

45. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 45 Main Scope: Unit–testing The same “assertion style” in different languages. Provides:- The name of the test case that failed. -The name of the source file that contains the test. -The line number where the failure occurred Different platforms/compilers: (Linux/Solaris/Windows) Test Frameworks: X-Unit family X-Unit family JUnit QtUnit CppUnit PyUnit A simple test: 1.Subclass the TestCase CppUnit or PyUnit class 2.Override the method runTest(). 3.When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expression that is true if the test succeeds

46. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 46 Test Frameworks: Oval Oval: validation and regression developed & used within CMS experiment Can be use for Unit-test to Validation tests. Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) It is possible to set different run environments. Can run external scripts and external binaries. Ovalfile (configuration) Executable Test Source Code Reference file Log file Oval run Oval diff Oval build

47. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 47 Test Frameworks: QMTest QMTest Uses a graphical interface for creating and running tests The configuration files are in XML and can be created from the GUI. We provide also script to do it Runs tests in parallel Organizes tests hierarchically Supports execution of a single test or many at once Records dependencies among tests Can be run in batch mode  easy integration with the Nightly-Build systems

48. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 48 Test Documents For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. We use a templatetemplate Test Case template collects information about the existing tests and how they cover the expected functionality of the software.

49. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 49 User support: How-To * * * * * 3 Follow sw-testing policies Automatic testing requires some standardization 1 2

50. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 50 Overview of LCG QA (slides by Jakub Moscicki) The main goal of QA activity is to help LCG projects assess and improve the quality of their software and procedures provide tools to collect useful metrics/statistics which help asses software quality; generate reports; verify if project setup is correct and compliant with LCG Policies.

51. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 51 QA Tools and Focus Reporting tools: lcg-qa-project-report.py analyze project tree in AFS release area time-based analysis (e.g. bugs reports)  generate HTML pages Release process tools – include all open/fixed bug reports in the release notes automatically Tests/Bugs are central for QA in our environment – vague/changing user requirements, no “product specifications” – tools/procedures by agreement rather than by decision – sophisticated code metrics LCG Policies – agreed and defined by AF – SPI supports them in the tools and procedures and only helps to work them out

52. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 52 QA Checklist Build the release Run automatic tests Statistics Test Inventory Documentation/Examples Inventory Savannah Statistics Code Inventory Rule Checker, Logiscope LCG Policies Configuration of a build system CVS directory structure Well-defined, transparent, open clear rules and checklist of assessed items anybody at anytime may see project statistics and create reports themselves anybody may contribute scram project SEAL SEAL_1_1_0 cd SEAL_1_1_0 cvs -d :pserver:anoncvs@lcgapp.cern.ch:/cvs/SEAL co -r SEAL_1_1_0 -d src seal eval `scram runtime -sh` qmtest -D./src/config/qmtest run -o test_results.qmr

53. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 53 QA Activities Manual / semi-automatic reports POOL QA for 0.4.0, 1.0.0, 1.1.0, 1.2.0 SEAL QA 0.3.1, 1.0.0 Development / integration of automatic tools SEAL_1_1_0 tools about to be released / announced Evaluation of tools Rule Checker LCG Coding Rules vs. existing activities (Atlas Rules) Logiscope, Test coverage SLOC, Valgrind, ignominy

54. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 54 Software Distribution Overview (slides by Jakub Moscicki) Temporary solution local installations (external sites, laptops,...) using simplest approach -python downloader + tar format -replicate the central AFS tree (in a optimized way) -package dependency from SCRAM...until a generic, long-term solution available SPI will adopt what LCG Grid Deployment will provide

55. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 55 Software Distribution users Simple tool to install: lcg-installation-manager.py quite successful for users: -POOL @ Karlsruhe -BNL nightly builds, CMS -developers at home, etc very easy to use and reliable Different use-cases should have different solutions Our tool is adequate as a temporary solution for LCG Application Area Distribution but long-term solutions must be investigated: -pacman, LCFGng.... GRID WN installations should be supported differently

56. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 56 LCG Soft Main Page

57. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 57 Project download page

58. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 58 Difficulties Technical difficulties are quite few Mostly human issues Normal as is the first time for most of users Difficult to adapt to an infrastructure Explain that developing infrastructure is to avoid for maintenance and quality issues Very large variety of projects and users Decisions are taken at last minute, but is normal in an active environment Everything needs to be discussed many times, which is good to get a good solution. Sometimes a bit too many Quite an unglamorous work but is useful to all projects

59. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 59 Achievements There is a set of services that are working Savannah Project Portal Software Testing External Software Service Quality Assurance and Policies Software Distribution Software Documentation Build and nightly builds …and more They are used by the LCG projects and outside the LCG Unlike in the past, we match the environment and habits of the way people work in HEP Basic services with open source software

60. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 60 Achievements (2) We have followed the strategy defined Work with the users Ask their help Develop as little as possible Provide simple and modular solutions We have commitments to the users but also to provide a sustainable service Is normal that most people move on to new projects, as it was planned Some people left SPI but the inter-personal relationships are very good and all still help Important achievement for me is that the project is a good group and works well with the projects people that genuinely help each other even after they are on other projects care for the service we provide and for the users

61. A. Aimar - EP/SFT LCG - Software Process & Infrastructure 61 Conclusions Homogeneity is important for the LCG and for its projects People can move from project to project or work across projects  common structure, same tools, etc Common services avoid duplication of work in the projects  software library, repository Users/developers get a uniform entry point to all projects  projects portal SPI’s address is  http://spi.cern.ch Nothing is specific to the LCG