1. INFSO-RI-223782 Support for IPv6 in ETICS EGEE’08 Conference, Istanbul, 22-26 September 2008 Marian ZUREK CERN - ETICS Marian.ZUREK@cern.ch

2. INFSO-RI-223782 Contents General overview Who/what we are: Partners/Architecture ETICS Infrastructure IPv6 compliance (IPv6 plugin) IPv6 Testbed Examples of usage ETICS plugin framework Future work Conclusions 2

3. INFSO-RI-223782 General Overview ETICS stands for E-infrastructure for Testing, Integration and Configuration of Software ETICS started in January 2006 and ended in December 2007. It had a very successful review exceeding all stated objectives ETICS 2 started in March 2008 (ranked first in the proposal selection process) and it’s supposed to run until February 2010 (mostly with EC funding) and then on its own ETICS is not ‘just’ a build system, it’s a complete infrastructure for building, testing, configuring and managing software projects 3

4. INFSO-RI-223782 Partners 4

5. INFSO-RI-223782 Architecture 5

6. INFSO-RI-223782 What ETICS is It’s a software engineering management system It’s a build and test infrastructure It provides tools and resources to configure, manage and analyse build and test runs It provides a common interface to diverse projects to facilitate knowledge sharing and operations management It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions It’s multi-platform and independent from any specific build or test tool 6

7. INFSO-RI-223782 ETICS Users EGEE uses ETICS within four Activities: NA4, JRA1, SA2, SA3 JRA1 and SA3 are the main ETICS users SA2 has implemented with ETICS the IPv6 compliance analysis of gLite code and the distributed IPv6 experimental testbed where the IPv6 version of BDII (and more) has been tested NA4 is using ETICS for experimenting with a few projects like Gridway, mpi, SAGA and the panc quattor compiler 7

8. INFSO-RI-223782 The ETICS Infrastructure The ETICS Infrastructure is currently based on three resource pools managed by Condor, at CERN, UoW and INFN There are about 450 cores available in the three sites with more than 40 different types of platforms At CERN in particular there are 120 cores and 12 different platforms (a platform is a combination of operating system, CPU architecture and compiler, ex: slc4_ia32_gcc36, deb4_x86_64_gcc412, etc) Additionally various nodes are attached to the CERN pool from third-party projects or organizations for specific use (4D Soft Ltd for DILIGENT in Hungary, GARR for EUChinaGrid and EGEE/SA2, IN2P3 for EGEE/SA2) 8

9. INFSO-RI-223782 The ETICS Infrastructure 9

10. INFSO-RI-223782 The CERN Resource Pool 10

11. INFSO-RI-223782 More about ETICS … Please join us on Thursday, the 25 th of September in the Malazgirt Hall between 16:00 and 19:00. 11

12. INFSO-RI-223782 IPv6 support in ETICS As a consequence of task defined in SA2/EGEE (EU requirement) for assuring grid interoperability integrate sites/countries lacking public IPv4 address ETICS has been contacted as a partner in this task 12

13. INFSO-RI-223782 Under the hood or what’s needed for IPv6 tests 13 A dedicated Condor configuration has been adopted to assure the proper match making for jobs requesting IPv6 resources (not visible to the user) host_network_stack = "IPv6” STARTD_EXPRS = $(STARTD_EXPRS) host_network_stack START = (job_network_stack =?= $(host_network_stack)) “extra” part in the submision engine (not visible to user) append_requirements = ( host_network_stack =?= "IPv6" ) ++job_network_stack = "IPv6”

14. INFSO-RI-223782 Is your code IPv6 compliant? 14 ETICS@PSM - 3 July 2008 For assessing the source code compliance a ETICS IPv6 Plugin has been developed and is present in the main-stream ETICS distribution Language coverage: Survey under preparation for the most common languages: C, C++, Java, Python, PERL Disclaimer: The passive check of the source code by the means of IPv6 plug-in presents assessment of the IPv6 compliance. Only the run-time tests could confirm the 100% compliance of your code. More on that - please see talks by Mario & Etienne. Please keep that in mind when watching coming slides.

15. INFSO-RI-223782 Is your code IPv6 compliant?

16. INFSO-RI-223782 Running tests on IPv6 enabled resources

17. INFSO-RI-223782 https://etics- repository.cern.ch:8443/repository/download/volatile/gli te_3_1_0/OverallReports/org.glite/org.glite/3.1.0- 1/slc4_x86_64_gcc346/reports.tar.gz$reports/ipv6/inde x.html Where to look for gLite IPv6 compliance reports 17

18. INFSO-RI-223782 IPv6 compliance report - ETICS

19. INFSO-RI-223782 IPv6 compliance report - gLite

20. INFSO-RI-223782 IPv6 compliance report - gLite/LCG-DM

21. INFSO-RI-223782 Plugins and Metrics Collectors The ETICS system is plugin-based (like for example ECLIPSE) It provides a core set of tools (like IPv6) and a published specification for developing additional plugins Plugins are essentially thin wrappers around existing or custom tools providing very specific functionality like packaging, static or dynamic testing, standards compliance testing, service installation and management, reporting, etc Plugins can publish information in the ETICS system in the form of metrics, which can then be used to do data mining or trend analysis using the available ETICS reporting tools Should another plugin be needed - it can be easily added to ETICS 21

22. INFSO-RI-223782 Examples of metrics collectors MetricsTypeProgramming languages/ technologies ToolETICS Plugin ComplexitystaticJava Python JavancssJCcnPlugin PyComplexityPlugin.py Design qualitystaticJavaJdependJDependPlugin N of potential bugsstaticC/C++ Python Perl PHP Java Flawfinder, RATS PMD Findbugs CFlawfinderPlugin CPyPhpRatsPlugin JPmdPlugin JFindbugsPlugin N of potential bugsdynamicC/C++ValgrindCValgrindPlugin Lines of codestaticAllSLOCCountSLOCCountPlugin CoveragedynamicJavaEmma Cobertura JUnitemmaPlugin JCoberturaPlugin Unit tests success rate dynamicJava Python JUnit PyUnit JUnitPlugin JUnitreportsPlugin.py PyUnitPlugin.py Compliance with standards staticIPv6 WSI IPv6Plugin WSInteroperabilityPlugi n ProfilingdynamicC/C++ Java Jrat Valgrind JRatPlugin CValgrindPlugin 22

23. INFSO-RI-223782 Fixing your code – what it takes 23 IPv6 related issues could be quickly and easily fixed JAVA applications are already IPv6 compliant (proper system parameters must be set) - examples: /proc/sys/net/ipv6/bindv6only if set to 1 will not accept IPv6 calls e.g. only changes in 9 (nine) lines were necessary to make BDII IPv6 compliant

24. INFSO-RI-223782 An example IPv6 programming: a dual stack server BDII The first version, an IPv6 server only, that opens one socket using IPv4 mapped address, is easier to develop: Two files and few lines were patched Only 4 lines in “bdii-fwd” were patched: [root@quarks sbin]# diff bdii-fwd.map../sbin.sav/bdii-fwd 10,11d9 < use Socket6; < use IO::Socket::INET6; # UREC/CNRS EGEE-SA2 xj 61c59 new(@proxy_server_config) > $proxy_server = IO::Socket::INET->new(@proxy_server_config) 169c167 new(@rs_config) > $remote_server = IO::Socket::INET->new(@rs_config) 24

25. INFSO-RI-223782 An example IPv6 programming: a dual stack server BDII Only 5 lines in “bdii-update”: [root@quarks sbin]# diff bdii-update.map../sbin.sav/bdii- update 17d16 < use IO::Socket::INET6; # UREC CNRS EGEE SA2 xj 416c415 new(@port2skip); > my $s = IO::Socket::INET->new(@port2skip); 445c444 new(@port2keep); > my $guard = IO::Socket::INET->new(@port2keep); 570c569 new(@port2skip); > my $s = IO::Socket::INET->new(@port2skip); 639c638 < system("$bdii_fwd_prog --local :: --service $bdii_port_read" > system("$bdii_fwd_prog --local 0.0.0.0 --service $bdii_port_read" 25

26. INFSO-RI-223782 Lesson learned Taking into account the site policies, building the Condor pool out of geographically dispersed Worker Nodes isn’t a trivial task as it may appear. Numerous issues have been identified and adressed: Reverse DNS: Solved by hard-coding the IP addresses in the Condor config files Firewall policies (procedures, transitional openings) Condor: Job Match-making: partial match was causing the ETICS production pool throughput degradation Power-cuts, etc. Mail, IM, SMS, phone were very useful 26

27. INFSO-RI-223782 Latest development I am leaving floor to Etienne who will be talking about extension to IPv6 code checker. We call it “LD_PRELOAD ipv6 checker”. The discussion will cover technique used and also summarise its applicability. 27

28. INFSO-RI-223782 Not enough? A dedicated tutorial is planned to happen during the JRA1 All hands meeting. For those not attending - please refer to our talks and references listed in handouts 28 ETICS@PSM - 3 July 2008

29. INFSO-RI-223782 Next Steps Keep the testbed operational Store working configurations and setup automated tests/deployments inside ETICS Make sure the ETICS service/infrastructure helps you in achieving your (IPv6 and other) goals (= provide continuous support to the community ) Your ideas – how would you like to proceed with your IPv6 compliance development and test we have a manually setup testbed – do we keep it operational or rather deploy the whole infrastructure automatically using ETICS Suggestions, ideas, complains ….. 29

30. INFSO-RI-223782 Conclusions ETICS in collaboration with GARR and UREC is providing the infrastructure enabling the community to perform the IPv6 tests Automated deployment mechanisms should be established to facilitate IPv6 compliance validation test We plan to continue the collaboration 30

31. INFSO-RI-223782 The End Q & A 31

32. INFSO-RI-223782 Benefits of Using ETICS Common set of interfaces and tools to different projects Simplifies working with different projects and transferring knowledge among technical people Configuration information is stored in a single place with a common schema and preserved while people or projects change Multi-platform support allow building and testing the same code-base on as many platforms as needed with negligible additional effort The resource infrastructure allows running several builds and tests without having to set up dedicated machines All machines are configured in the same way or use virtual images in order to guarantee reproducibility of results Standard build and test tools are available out-of-the-box and can be applied in the same way to different build/test runs and different projects over time 32

33. INFSO-RI-223782 Benefits of Using ETICS (2) The built-in packaging system abstracts the package information and automatically builds the appropriate package format for the different platforms (tarballs, RPMS, debs, MSIs, etc). No need to maintain separate scripts Reporting tools are available out-of-the-box to control the software evolution, pinpoint issues and quickly solve problems APIs and plugins allow to integrate the tools into existing development processes and extend them with custom tests or reports as needed New plugins and tools developed by a project can be of benefit to all other projects (no duplication of effort) ETICS provides a large set of external libraries in source format or already precompiled for many platforms (the ‘externals’ project). Components can set dependencies on these libraries and automatically configure their workspace. 33

34. INFSO-RI-223782 The ‘Distributed Testing’ Feature One of the last features to be added, still in experimental mode It allows designing complex tests involving several services and test applications to be deployed on separate nodes ETICS analyses the definition and deploys the services on the necessary nodes A synchronization mechanism orchestrate the start/stop of services and the execution of the test applications At the end the results are collected and reported as a single report It is not yet usable by ‘any user’, it requires some deep knowledge of the system to be tested It has been successfully used in a number of cases by the DILIGENT project It has the strong prerequisite that the services to be deployed have to be managed without user intervention, which is not always the case 34

35. INFSO-RI-223782 How to fix your code Small hint here …. This will be covered in a short tutorial by Mario Another presentation will be given during JRA1 All Hand meeting in …. dates/place TBConfirmed 35