1. A. Aimar - EP/SFT LCG - Software Process & Infrastructure1 SPI Infrastructure for LCG Software Projects Overview A.Aimar EP/SFT CERN LCG Software Process & Infrastructure

2. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 2 Project context of LCG SPI LCG Application Area LCG Infrastructure Common services Similar ways of working (process) Tools, templates, training General QA, tests, integration, release LCG Application Area software projects POOL: Persistency SEAL: Core common software PI: Physics Interfaces Simulation Detector Description …etc… LHC grid software applications (LHC experiments, projects, etc) LCG SPI project

3. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 3 Project context of the LCG SPI “Software Management Process RTAG” General recommendations -All LCG projects must adopt the same set of tools, standards and procedures -Adopt commonly used open-source or commercial software where available -Avoid “do it yourself solutions” -Avoid commercial software that may give licensing problems If each project needs an infrastructure, many projects need it even more… -Tools, standards and procedures -Try to avoid complexity

4. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 4 Infrastructure Software Development Software development Support Keep it “simple and modular” approach a. Provide general services needed by each project -CVS repository, Web Site, Software Library -Mailing Lists, Bug Reports, Collaborative Facilities b. Provide solutions specific to the software phases -Tools, Templates, Training, Examples, etc. Coding Analysis and Design Development Release Specifications Testing ….. Deployment and Installation ….. Planning

5. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 5 Project guidelines No specific software development The project should avoid any new development, and future maintenance, of any specific software package needed for the LCG software infrastructure. Use HEP or free software All software used should be already available, or become available, in the HEP community or in the free software community The goal of the project is to define an infrastructure and later a simple process Future maintenance will need separate planning and resources. Everything is done in collaboration LCG and LCG projects (Pool) IT division and LHC experiments Big projects (G4, Root, etc) Using as the existing IT services The project should use all existing IT services Make sure they match the LCG needs

6. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 6 http://spi.cern.ch

7. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 7

8. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 8 OUTLINE: SPI Services Services specific to LCG projects General overview Very useful to LCG project but not really to other projects Services for LCG projects but available External software Testing framework General service for LCG and others Project Portal

9. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 9 >>> CVS repository and Delivery Areas CVS repository A central CVS repository managed and available to all projects Any project just needs to ask for it, and declare its users permissions Managing mirroring and backups Users access controlled Tools for automatic clean up of locks, etc Systems to browse the repositories Delivery areas The AFS and DFS delivery areas 1.an area to install software created by projects in the LCG application area 2.an area where external and third party software 3.an area where software is installed which is provided by people for evaluation within a project Pre-installed software shall be in directories specifying the package name, the package version and the “OS_Compiler”

10. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 10 :pserver: @lcgapp.cern.ch:/cvs/… \\cern.ch\dfs\projects\lcg /afs/cern.ch/sw/lcg

11. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 11 >>> Coding guidelines Rule Checker (on the way) Human Reviews from SPI

12. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 12 >>> CVS organization

13. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 13 >>>Scram Used by Pool and Seal Improvements restructuring the build system to improve performance possible to use automake/autoconf within a scram project Porting to Windows Execution on perl on Windows and usage of nmake (standard Windows installation without cygwin) Import projects into Visual Studio Organization a single SPI-managed toolbox service for LCG projects Organizing a tutorial and training for LCG developers and users

14. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 14 >>> LCG Workbook http://lcgapp.cern.ch/project/workbook/

15. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 15 Other components available Memory Leaks Automatic Builds (on the way) Software distribution (investigating “pacman”) Software documentation, templates, etc

16. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 16 >>> Code Documentation Features of interest: Code browsing Code searching Code information Design diagrams Any project will have them available as part of the infrastructure Doxygen LXR ViewCVS

17. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 17 Code documentation: Doxygen

18. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 18 Code documentation : LXR

19. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 19 Code documentation: ViewCVS CVSgraph: Displays the tree of revisions and branches graphically Enscript: Colorize files in the CVS repository.

20. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 20 >>> External Software Service

21. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 21 Service: Software Library All software needed by more than one LCG projects shall be installed in common areas Available for several compilers and operating systems Maintained centrally with support and help to the user External documentation available Contact with the authors if the software is developed in HEP or LCG Distribution and pre-installation to all LCG and HEP users

22. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 22 /afs/cern.ch/sw/lcg/external

23. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 23 Current status Central installation of HEP or open source software needed by LCG related projects or following their requests in term of version and platforms (linux, MSWindows and Solaris when needed). Standard installation & location Availability AFS: /afs/cern.ch/sw/lcg/external/ DFS: \\cern.ch\dfs\project\lcg\external\ (just started) sundev005 /afs/cern.ch/user/l/lcgspi [87]> ls /afs/cern.ch/sw/lcg/external/ Ant/ JACE/ Python/ edg-rls-client/ png-lib/ valgrind/ Boost/ JDK/ TJDO/ expat/ root/ wxPython/ CMake/ JDORI/ XalanJ/ gccxml/ scripts/ Colt/ Log4J/ XercesC/ mysql/ tarFiles/ CppUnit/ MMMySQL/ XercesJ/ mysql++/ test.html* GSL/ Oval/ doxygen/ packages/ uuid/

24. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 24

25. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 25

26. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 26

27. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 27

28. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 28 >>> Software Testing Tests of classes individual class small groups of collaborating classes work package Software developer CODE Test programs Automatic software test run process Code Documentation Tools CppUnit PyUnit Oval

29. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 29

30. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 30 Test frameworks: CppUnit A simple test 1.Subclass the TestCase CppUnit class 2.Override the method runTest(). 3.call CPPUNIT_ASSERT Similar tools: Junit, PerlUnit, PyUnit, QtUnit Output in XML, compiler or text Windows version for MVC++6.0

31. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 31 Test frameworks: CppUnit

32. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 32 Test frameworks: Oval Ovalfile (configuration) Executable Test Source Code Reference file Log file Oval run Oval diff Oval build Can be use for Unit- test It is possible to set different run environments Can run external scripts and external binaries Authors modified it to run it with or with any environment Oval: validation regression

33. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 33 Test frameworks: Oval

34. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 34 Test: structure and documentation SPI-TESTING component SPI-CVS_structure component CVS testing structure Test-naming  Will help to have a automatic way to run test code Test documentation Test cases specification template srcdoctest testAtestB Ovalfile Nametest.ccp Nametest.ref Nametest.log [Nametest.in] [Mametest.out]

35. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 35 Testing Available CppUnit, PyUnit and Oval Can also be used stand alone, integrated with Scram and with CMT Can also be used by projects NOT using Scram

36. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 36 >>> Project Portal Bug Tracker A central installation so that each project has an available bug tracker system Centrally managed but customizable for different needs Easier to post bug if projects use similar systems No work of maintenance and upgrade on the projects Web based http://lcgappdev.cern.ch (for a few test projects) Project Portal The development front-end of every project With all features needed, configured and integrated News and Faq Task management Mailing lists access Bug tracking access Download area Access to the CVS repository We have customized “Savannah” from GNU

37. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 37

38. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 38 Benefits Totally web based Single entry point to project(s) Uniform access to project information Set up common web infrastructure for a project without coding

39. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 39

40. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 40 Bug tracker Fields customizable for each project File uploads Audit trail Full text searches Cc forwarding

41. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 41

42. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 42 News system Messages can be displayed in threads Monitoring (automatic sending of email) Link detection Transforms URLs to links References to bugs/tasks get linked

43. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 43

44. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 44 Task manager Sub-project/task hierarchy Tasks can be assigned to developers Tasks carry a deadline, etc. Tasks/bugs interdependencies

45. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 45

46. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 46 Personalize web page A logged in user gets a customizable personal entry page display of current work status -User's bugs, tasks,... -Monitored news threads -Bookmarks

47. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 47

48. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 48

49. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 49 And much more… Consolidate project links ViewCVS (, Doxygen, LXR, Bonzai,...) Project homepage Mailing lists (subscribe/unsubscribe) FAQ system (user/admin documents) Support manager Download area Patch manager Projects administrators can choose what to use for each project

50. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 50 Admin: Project types Name and description Preferences common to one type of projects Homepage, CVS information List of status descriptions (e.g. alpha, beta, stable) Download/upload area (faq) information Allow use of other subsystems (patch manager, task manager, news, support request manager, bug tracker, FAQ)

51. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 51 Admin: For each project Public information -Name, description, development status -Enable subsystems ( bug tracker, patch manager, task manager, CVS link, news system, support tracker, FAQ system, download/upload area ) Members Task manager, patch manager, support manager: none, tech, admin tech&admin Project administrator

52. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 52

53. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 53 Admin: Bug tracking configuration Bugs -Fields, values -Email notification -Global: List of 'external' email addresses who get all new bug submissions -Personal (techs+admins): List of 'Users to watch' -Personal(techs+admins): Detailed list of events that trigger email to you News Delete postings

54. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 54

55. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 55

56. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 56 Specific implementations General bug fixing and improvements Implemented bulk user registration Get a list of users and add/remove them all without registering them one by one Integration with AFS authentication Added a PAM module to verify a login against the AFS authentication The user can still define a password not using the AFS password Sending back to Savannah our changes, patches and fixes Run a stable service http://lcgappdev.cern.ch for now

57. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 57 Proposed release schedule December 2002 - Beta Versions Many components will be available before February 2003 - Version 1.0 All components and services mentioned above except Software Library (but basic), Project portal (but basic) Monitoring if resources will need to be injected May 2003 - LCG Test - Version 1.1 Software Library, Project portal and all features Standard templates and tools available September 2003 - Version 1.2 LCG software process Standards for user specifications Design guidelines and specifications end of 2003 until 2005 Development Version 2, support and maintenance

58. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 58 Conclusion Homogeneity is important for the LCG and for its projects All work is done seeing the experts and work in LHC experiments and LCG projects We all need to compromise and NOT focus on infrastructure and individual taste The reference URL is http://spi.cern.ch the others in this presentation may change.

59. A. Aimar - EP/SFTLCG - Software Process & Infrastructure 59 http://spi.cern.ch