1. Promoting and Standardizing Grid Computing Grid and Distributed Software Certification and Quality Assurance GGF17 Certification BoF Marc-Elian Bégin CERN

2. Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo2 Agenda Introduction ETICS overview OMII (Steven Newhouse) Discussion Conclusion

3. www.eu-etics.org INFSOM-RI-026753 ETICS: E-infrastructure for Testing, Integration and Configuration of Software Marc-Elian Bégin Dissemination, certification and web tools Work- package Leader CERN

4. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 4 Vision and Mission Vision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applicationsVision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applications Mission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software. Set up the foundations for a certification process to help increasing the quality and interoperability of such softwareMission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software. Set up the foundations for a certification process to help increasing the quality and interoperability of such software

5. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 5 Short Term Use Cases Automatically build software on different platformsAutomatically build software on different platforms Automatically collect and store metrics during build processAutomatically collect and store metrics during build process Quality Assurance trend analysis on collected metricsQuality Assurance trend analysis on collected metrics Automatic handling and management of dependenciesAutomatic handling and management of dependencies Automatic execution of testsuites, including build, installation and configuration of dependenciesAutomatic execution of testsuites, including build, installation and configuration of dependencies “Shop” dependencies from quality measured artifacts (components, services and plug-in) via managed repository“Shop” dependencies from quality measured artifacts (components, services and plug-in) via managed repository Offload deployment of complex middleware servicesOffload deployment of complex middleware services Combinations of the aboveCombinations of the above

6. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 6 ETICS in a Nutshell ETICS stands for e-Infrastructure for Testing, Integration and Configuration of SoftwareETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software It’s an SSAIt’s an SSA It has been granted a contribution of 1.4 M€It has been granted a contribution of 1.4 M€ It has a duration of two yearsIt has a duration of two years The project has started on January 1 st, 2006The project has started on January 1 st, 2006

7. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 7 The ETICS Partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT The Condor batch system, distributed testing tools, service infrastructure, NMI

8. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 8 ETICS Objectives Objective 1 (technical)Objective 1 (technical) –Provide a comprehensive build and test service especially designed for grid software –Support multi-platform, distributed operations to build software and run complex test cases (functional, regression, performance, stress, benchmarks, interoperability, etc) Objective 2 (coordination, policies)Objective 2 (coordination, policies) –Establish the foundations for a certification process –Contribute to interoperability of grid middleware and applications by promoting consistent build and test procedures and by easying the verification of compliance to standards –Promote sound QA principles adapted to grid environment through the participation to conferences, workshops, computing training events (CSC, ICEAGE, EGEE)

9. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 9 Service Overview Build/Test Artefacts Web Application Report DB Project DB NMI Scheduler Clients Web Service NMI Client Wrapper Via browser Via command- Line tools WNs ETICS Infrastructure

10. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 10 User roles Guest: can execute local builds and test of modulesGuest: can execute local builds and test of modules Developers: can create configurations and execute local and remote build and test of modulesDevelopers: can create configurations and execute local and remote build and test of modules Integrators: can register build artefacts in repositoryIntegrators: can register build artefacts in repository Testers: can register and perform trend analysis of test resultsTesters: can register and perform trend analysis of test results Release Managers: can publish and release artefacts, create release notes and other documentationRelease Managers: can publish and release artefacts, create release notes and other documentation Module Administrator: create modules and assign permissions with a given projectModule Administrator: create modules and assign permissions with a given project Administration: can create and manage projects, assign permissions, set policies, etcAdministration: can create and manage projects, assign permissions, set policies, etc

11. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 11 QA Certification Process It’s the other major goal of the projectIt’s the other major goal of the project ETICS must deliver a feasibility study for the implementation of a QA certification process for grid and distributed softwareETICS must deliver a feasibility study for the implementation of a QA certification process for grid and distributed software Not just a document, but a full assessment of what procedures, tools and rules maximize the chances of project of delivering reliable, interoperable software based on the real data collected in two years of activityNot just a document, but a full assessment of what procedures, tools and rules maximize the chances of project of delivering reliable, interoperable software based on the real data collected in two years of activity We will propose it as the starting point of a coordinated QA Certification activity in the context of a more permanent grid infrastructure initiativeWe will propose it as the starting point of a coordinated QA Certification activity in the context of a more permanent grid infrastructure initiative Will be done in close collaboration with the ETICS Experts Board to make sure it is practical and realisticWill be done in close collaboration with the ETICS Experts Board to make sure it is practical and realistic

12. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 12 Standardisation effort e-Infrastructure Reflection Group (e-IRG)e-Infrastructure Reflection Group (e-IRG) –e-IRG mandate: “coordinates on a high European level the introduction of a (grid based) infrastructure for e-Science” –Working on raising awareness in –Software Quality Assurance and automation –Standard software Certification Process –Co-chaired a working session during last e-IRG meeting in Linz on “Middleware Interoperability and Repository” Global Grid Forum (GGF)Global Grid Forum (GGF) –GGF mandate: “international community leading the pervasive adoption of grid computing in research and industry” –Chairing a working session in the next GGF meeting in Tokyo on “Grid and Distributed Software Certification and Quality Assurance” –Raise awareness on the need for Standard Certification Processes, including automated tests for interoperability

13. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 13 Typical ETICS working scenario OMII EU/UK HEP \ BIO \ INDUSTRY ETICSOMII EU/UK Repositories Other (project specific)

14. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 14 Repositories All artifacts produced when using the ETICS services are made available to the community at largeAll artifacts produced when using the ETICS services are made available to the community at large –Source and binary packages –Test reports –Quality assurance reports –Benchmarks –Interoperability evaluations, etc ETICS is involved in digital libraries and repositories on two fronts:ETICS is involved in digital libraries and repositories on two fronts: –ETICS has among its partners DILIGENT, a Grid powered Digital Library –ETICS is collaborating with the EU-OMII project to set up a reference repository of quality assured software packages

15. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 15 Virtual Test Lab The Grid must embrace diversity in terms of platforms, development languages, deployment methods, etcThe Grid must embrace diversity in terms of platforms, development languages, deployment methods, etc However, testing all resulting combinations is very difficult and time consuming, not to mention the manpower requiredHowever, testing all resulting combinations is very difficult and time consuming, not to mention the manpower required ETICS is proposing a high-efficiency automation process, but still providing and especially maintaining the required hardware and software resources is not trivialETICS is proposing a high-efficiency automation process, but still providing and especially maintaining the required hardware and software resources is not trivial A possible solution is the use of virtualisationA possible solution is the use of virtualisation

16. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 16 Virtual Test Lab ETICS Engine

17. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 17 The Virtual Test Lab ETICS Engine Microsoft Virtual Server Images can contain preinstalled OSs in fixed, reproducible configurations Security risks are minimized, the image is destroyed when the test is over. But it can also be archived for later offline analysis of the tests

18. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 18 Example of Web Application Project structure

19. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 19 Example of Web Application Project configuration

20. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 20 Example of Build Report

21. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 21 Example of Test Report

22. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 22 Example of Job Submission Report Preliminary integration of the web application with NMI

23. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 23 Example of Client

24. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 24 Status – ETICS Development ClientClient –Available for local and remote building and testing software (read only) –Editing feature under development ServerServer –Available for building and testing software (read only) –Editing feature under development –Secure access under development –Reports engine under development Web ApplicationWeb Application –Available for browsing ETICS metadata –Editing feature under development Editing stopgap solutionEditing stopgap solution –Tool for registering new projects

25. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 25 Status – Usage Build systemBuild system –Large element of gLite 3.0 building –On track to build entire gLite 3.1 stack –In the process of building CERN/FIO Quattor software –In the process of building DILIGENT middleware/software –Building ETICS software Test systemTest system –Porting gLite Certification Testsuites –Executing ETICS stress testsuite DeploymentDeployment –UoW: ~70 machines –CERN: ~20 machines –INFN: ~3 machines

26. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 26 Potential new users Automatic execution of build/testAutomatic execution of build/test –MPI testsuites execution and trend analysis: requested by Cal Loomis (EGEE/NA4) –Build and test of OMII components (UK+Europe) CERN Administrative Information ServicesCERN Administrative Information Services –Build and test of J2EE applications OthersOthers

27. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 27 Long Term Future and Sustainability We envision ETICS to become a permanent service after its initial two-year phaseWe envision ETICS to become a permanent service after its initial two-year phase As projects start using and relying on it for managing the software development cycle, the ETICS infrastructure should get enough “critical mass” to be supported by research and industrial organisation as other “commodity” servicesAs projects start using and relying on it for managing the software development cycle, the ETICS infrastructure should get enough “critical mass” to be supported by research and industrial organisation as other “commodity” services In addition, we want to propose ETICS as one of the cornerstones of a more permanent international collaboration to establish a European and world-wide grid infrastructureIn addition, we want to propose ETICS as one of the cornerstones of a more permanent international collaboration to establish a European and world-wide grid infrastructure

28. INFSOM-RI-026753 Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 28 Thanks http://www.eu-etics.org

29. Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo29 OMII

30. Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo30 Questions/Issues for discussion How do you see certification and QA in your context? −Do you need it? −Are you doing it? −If you don’t, why? −Would a “standardised certification process” be useful? Building −Do you do multi-platform build and test? −If so, how? −If not, who does it for you, or why not? −Where do you put your build artefacts? −Where do you find your dependencies and how to you manage them? −How do you manage your releases? Testing −Do you have testsuites? Are they automated? −Are you using tools to implement and execute your tests? −How do you execute them and how do you handle test outputs? −How do you analyse the results Scope for standardisation −Process −Test execution and output formats −Metrics and their collection