EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks GLite testing status and future Gianni Pucciani EGEE 2008, Istanbul 24 September 2008

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 2 Outline  Introduction.  Current status of the testing process.  Running tests and regression tests.  Writing new tests, guidelines.  The certification Testbed (slides from Louis Poncet) and the vNode portal.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 3 Testing activity What we test: –Deployment + functional + regression tests on the CTB, during patch certification; to test the service functions using API or CLI, including basic service checks (ping test, BDII checks). –Performance and stress tests done occasionally on dedicated machines (e.g. CREAM CE tests from Di Qing What we do not test: –Unit tests, must be done by service developers. –Complex and systematic use cases, performance and stress tests done on request during “pilots” in pre-production.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 4 Current status List of available tests: Service certification checklist For patch certification tests are run from the command line (Shell, Python, Perl scripts). Meta-script are available whenever possible to run all the necessary test scripts in one shot. A framework to automatically run regression tests is available. Writing tests is a continually evolving process; tests must be easily added and integrated in the test repository and made immediately available for patch certification.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 5 Current status: testing during patch certification Testing during patch certification: 1. Install a (virtual) machine with the given service. 2. Reproduce the bug. 3. Install the patch. 4. Verify that the bug has been fixed. 5. Follow the instructions on the service certification checklist on how to run tests for the given service. Usually checkout the tests from CVS and launch a bash meta-script. When available, run regression tests using the regression tests framework. 6. Attach the meta-script and regression tests output to the Savannah patch tracking system.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 6 Current status: testing scripts, CVS architecture Tests scripts are available on CVS: ( Example: AMGA/ tests/ AMGA-test_ping.py AMGA-test_functions.py AMGA-test_statistics.py manual/ README AMGA-certtest.sh AMGA-certconfig.sh APEL/ BDII/

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 7 Current status: running the tests #>./AMGA-certtest.sh START Mon Sep 15 14:10:57 CEST Executing AMGA-test_ping.py AMGA-test_ping.py PASSED Executing AMGA-test_functions.py AMGA-test_functions.py PASSED Executing AMGA-test_statistics.py AMGA-test_statistics.py PASSED END Mon Sep 15 14:11:01 CEST 2008 TEST_PASSED #>./AMGA-certtest.sh START Mon Sep 15 14:09:50 CEST Executing AMGA-test_ping.py AMGA-test_ping.py PASSED Executing AMGA-test_functions.py AMGA-test_functions.py FAILED Executing AMGA-test_statistics.py AMGA-test_statistics.py PASSED END Mon Sep 15 14:09:54 CEST 2008 TEST_FAILED The following tests failed: AMGA-test_functions.py: results in tests/AMGA-test_functions.py_result.txt Example from the AMGA tests:

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 8 Regression tests A regression test is bound to a specific bug, and it may involve the deployment of multiple nodes. A number of regression tests (mainly DM and WMS) is already available. A framework to easily run tests and extend the regression tests repository is available: Certifiers are strongly encouraged to write regression tests; during patch certification it often requires a minimal overhead. From our experience more than 50% of the bugs are candidates for regression tests.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 9 Regression test writing./regTest.sh./bugs/ bug1596 bug22165 …./config/ commonFunctions.sh config.sh./testlists/ mybugs.txt mybugs.sh./regTest.sh –tl testlists/mybugs.txt … VO=dteam SE_DPM=se.cern.ch … test_bug1596_pre () test_bug1596 () test_bug1596_post () by Andreas Unterkircher SA3 All hands meeting May 2008

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 10 Missing tests A list of missing tests is available and constantly updated: Some examples: –File Transfer Service (FTS):  Testing the FTS Java API.  Further tests with all combination of SEs.  Tests with different VOMS roles. –LCG File Catalog (LFC):  Test the Perl API.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 11 Writing new tests If you want to collaborate in writing tests you are more than welcome to contact me, a task will be assigned to you and tracked through the Savannah task tracking system. Test writing guidelines: –Provide a test plan explaining the tests that will be implemented. –Priority to service ping tests and functionality tests (CLI and API). –Tests should be launched from the command line (Bash, Perl, Python), without the need of using any framework. The CERN certification team will take care of the integration in the test repository. –Provide adequate documentation. –Tests maintenance is done by the CERN certification team, with the test developer’s help when needed.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 12 Testing frameworks Testing during patch certification has to be quick and easy so that we can profit from a distributed and dynamic list of certifiers provided by SA3 partners. Testing frameworks: –SAM: is currently used to monitor the certification Testbed. –Nagios: under evaluation. It will be used to monitor the certification Testbed. –ETICS: under evaluation, so far more suitable for unit tests and deployment tests (rpm installation) done by developers.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 13 Conclusions What we test, what we don’t and why. Tests are developed by SA3 partners, the CERN certification team integrates them in the test repository and provides meta-script when possible/necessary. To make certifiers’ life easier we provide: –Service certification checklist. –Tests maintenance. –Regression tests framework. –Certification Testbed. Providing a full set of up-to-date tests for each component of the middleware is a continuous effort, have a look at the missing tests list and contact me to help us improve the situation.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 14 The Certification Testbed A little update about the certification Testbed Slides provided by Louis Poncet

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 15 Architecture Partners sites CE SE LFC BDII PX WMS WN Top BDII Physical TB Full VM testbeds User Interface

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, 16 vNode: Virtual Nodes On Demand Our virtualization portal based on the Xen hypervisor, maintained by Ricardo Mendes Web application: Web site: Accessible from within CERN, allow us to bring up machines with a few clicks for patch certification (32 or 64 bits, SLC4 or SLC3), destroying the machine and releasing the resources after the patch has been certified. Virtual grid: automatic deployment of Grid nodes with the possibility to select node type and configuration parameters for each virtual machine. This functionality is currently under development.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 17 Certification testbed infos The certification TB is now a full operational service under SLC4 DPM, LFC and WN on 64bits All machines are equivalent to production nodes –8 cores 64bit, 16 Gig of RAM, 160 Gig HD (RAID 5) All info to extend our TB with external site – Connection of the TB to VM SA3 service Security centrally manage trough the GD tool Monitoring of security package up-to-date : pakiti User access based of AFS at CERN The certification TB is a GRID

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 18 TB in CERN UIBDII site and top lcg CECream CE Torque serverWMS/LB MONVOMS DPM + DPM Pool (64 bits)LFC (64 bits) classic SE2 WNs (32bits & 64bits) FTSPROXY File serverMonitor : SAM + NAGIOS... Goal : To be able to give a reliable site, up-to-date with the latest certified patches. Can be used as base for your services.

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 19 Services provide by partners SITE NAMESITE SPECIALITY CESGASun Grid Engine PICCondor GRNETTorque (separate server) MONVOMS DESYDcache INFNLSF CYPRUSAMGA CESGASun Grid Engine

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 20 Security WE CAN’T IGNORE SECURITY AT ALL We are not stable (patch in certification, not certified) Apply all basics recommended by your security officer in your IT department and: –Advanced root authentication (no shared password), kerberos, ssh keys... –Only useful packages installed (don’t give the knife that will stab you). –Protect all authentication data (ssh keys, CA...). –Try to be in auto-update  Security package automatically install.  Gives early warning if OS updates will disturb the middleware. –Firewall as much as possible. –Intrusion detection system is good. –syslog server (to keep your logs safe). We have to share good idea and experience

Enabling Grids for E-sciencE EGEE-III INFSO-RI EGEE08 Istanbul, Author L.Poncet, Speaker G.Pucciani 21 Conclusion Need feedback from you, to increase the quality of service. Increase your security level : –We are really visible. Send your questions to :

Enabling Grids for E-sciencE EGEE-III INFSO-RI Useful links SA3 Testing and Certification –Available tests –Service Certification Checklist –Test writing guidelines –Missing tests –CVS tests repository Certification Testbed –Contacts and how to join the Testbed vNode –Web Site –Web Application