GLite 26.07.2006D. Zaborov, SA1-SA3 meeting, Dubna1 Dmitry Zaborov (SINP-MSU-Moscow) Dubna, Jul 26, 2007 Testing 3.0 under SLC 4.

Slides:



Advertisements
Similar presentations
Participation in the development of certification tests for LCG/GLITE Galaktionov V.V. The works presented in the report are executed in accordance to.
Advertisements

Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.
Development of test suites for the certification of EGEE-II Grid middleware Task 2: The development of testing procedures focused on special details of.
Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Services Abderrahman El Kharrim
Server-Side vs. Client-Side Scripting Languages
Basic Grid Job Submission Alessandra Forti 28 March 2006.
Guide to Linux Installation and Administration, 2e1 Chapter 6 Using the Shell and Text Files.
Automated Tests in NICOS Nightly Control System Alexander Undrus Brookhaven National Laboratory, Upton, NY Software testing is a difficult, time-consuming.
Basic Unix Dr Tim Cutts Team Leader Systems Support Group Infrastructure Management Team.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
M. Taimoor Khan * Java Server Pages (JSP) is a server-side programming technology that enables the creation of dynamic,
The ATLAS Production System. The Architecture ATLAS Production Database Eowyn Lexor Lexor-CondorG Oracle SQL queries Dulcinea NorduGrid Panda OSGLCG The.
Don Quijote Data Management for the ATLAS Automatic Production System Miguel Branco – CERN ATC
Nightly Releases and Testing Alexander Undrus Atlas SW week, May
9 Chapter Nine Compiled Web Server Programs. 9 Chapter Objectives Learn about Common Gateway Interface (CGI) Create CGI programs that generate dynamic.
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
How to Install and Use the DQ2 User Tools US ATLAS Tier2 workshop at IU June 20, Bloomington, IN Marco Mambelli University of Chicago.
LCG Middleware Testing in 2005 and Future Plans E.Slabospitskaya, IHEP, Russia CERN-Russia Joint Working Group on LHC Computing March, 6, 2006.
1 / 22 AliRoot and AliEn Build Integration and Testing System.
5 - 1 Copyright © 2006, The McGraw-Hill Companies, Inc. All rights reserved.
Enabling Grids for E-sciencE EGEE-III INFSO-RI Using DIANE for astrophysics applications Ladislav Hluchy, Viet Tran Institute of Informatics Slovak.
CERN-PH-SFT-SPI August Ernesto Rivera Contents Context Automation Results To Do…
Enabling Grids for E-sciencE Introduction Data Management Jan Just Keijser Nikhef Grid Tutorial, November 2008.
Jan 31, 2006 SEE-GRID Nis Training Session Hands-on V: Standard Grid Usage Dušan Vudragović SCL and ATLAS group Institute of Physics, Belgrade.
CERN Using the SAM framework for the CMS specific tests Andrea Sciabà System Analysis WG Meeting 15 November, 2007.
T3 analysis Facility V. Bucard, F.Furano, A.Maier, R.Santana, R. Santinelli T3 Analysis Facility The LHCb Computing Model divides collaboration affiliated.
INFSO-RI Enabling Grids for E-sciencE gLite Data Management and Interoperability Peter Kunszt (JRA1 DM Cluster) 2 nd EGEE Conference,
SRM Monitoring 12 th April 2007 Mirco Ciriello INFN-Pisa.
14th Oct 2005CERN AB Controls Development Process of Accelerator Controls Software G.Kruk L.Mestre, V.Paris, S.Oglaza, V. Baggiolini, E.Roux and Application.
FRANEC and BaSTI grid integration Massimo Sponza INAF - Osservatorio Astronomico di Trieste.
1 LHCb File Transfer framework N. Brook, Ph. Charpentier, A.Tsaregorodtsev LCG Storage Management Workshop, 6 April 2005, CERN.
1 Andrea Sciabà CERN Critical Services and Monitoring - CMS Andrea Sciabà WLCG Service Reliability Workshop 26 – 30 November, 2007.
DPM Python tools Ivan Calvet IT/SDC-ID DPM Workshop 10 th October 2014.
INFSO-RI Enabling Grids for E-sciencE Introduction Data Management Ron Trompert SARA Grid Tutorial, September 2007.
Dispatching Java agents to user for data extraction from third party web sites Alex Roque F.I.U. HPDRC.
G.Govi CERN/IT-DB 1 September 26, 2003 POOL Integration, Testing and Release Procedure Integration  Packages structure  External dependencies  Configuration.
SAM Sensors & Tests Judit Novak CERN IT/GD SAM Review I. 21. May 2007, CERN.
VO Box Issues Summary of concerns expressed following publication of Jeff’s slides Ian Bird GDB, Bologna, 12 Oct 2005 (not necessarily the opinion of)
Service Availability Monitor tests for ATLAS Current Status Tests in development To Do Alessandro Di Girolamo CERN IT/PSS-ED.
Liudmila Stepanova SINP-MSU team Testing gLite Worker Nodes LCG Dubna, Jul 26, 2007.
Pavel Nevski DDM Workshop BNL, September 27, 2006 JOB DEFINITION as a part of Production.
Lab 9 Department of Computer Science and Information Engineering National Taiwan University Lab9 - Debugging I 2014/11/4/ 28 1.
INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.
D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.
SAM Status Update Piotr Nyczyk LCG Management Board CERN, 5 June 2007.
JRA1 Meeting – 09/02/ Software Configuration Management and Integration EGEE is proposed as a project funded by the European Union under contract.
1 DIRAC Data Management Components A.Tsaregorodtsev, CPPM, Marseille DIRAC review panel meeting, 15 November 2005, CERN.
INFSO-RI Enabling Grids for E-sciencE File Transfer Software and Service SC3 Gavin McCance – JRA1 Data Management Cluster Service.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America LFC Server Installation and Configuration.
Open Science Grid Configuring RSV OSG Resource & Service Validation Thomas Wang Grid Operations Center (OSG-GOC) Indiana University.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Maria Alandes Pradillo, CERN Training on GLUE 2 information validation EGI Technical Forum September 2013.
Implementation of GLUE 2.0 support in the EMI Data Area Elisabetta Ronchieri on behalf of JRA1’s GLUE 2.0 Working Group INFN-CNAF 13 April 2011, EGI User.
SRM v2.2: service availability testing and monitoring SRM v2.2 deployment Workshop - Edinburgh, UK November 2007 Flavia Donno IT/GD, CERN.
Jean-Philippe Baud, IT-GD, CERN November 2007
Unit Testing.
Classic Storage Element
NA4/medical imaging. Medical Data Manager Installation
Patricia Méndez Lorenzo ALICE Offline Week CERN, 13th July 2007
Chapter 2: System Structures
GLAST Release Manager Automated code compilation via the Release Manager Navid Golpayegani, GSFC/SSAI Overview The Release Manager is a program responsible.
Leanne Guy EGEE JRA1 Test Team Manager
Chapter 27 WWW and HTTP.
EGEE Middleware: gLite Information Systems (IS)
Chapter 2: Operating-System Structures
Chapter 2: Operating-System Structures
Site availability Dec. 19 th 2006
Presentation transcript:

gLite D. Zaborov, SA1-SA3 meeting, Dubna1 Dmitry Zaborov (SINP-MSU-Moscow) Dubna, Jul 26, 2007 Testing 3.0 under SLC 4

gLite D. Zaborov, SA1-SA3 meeting, Dubna2 Overview Testing gLite 3.0 UI –A test suite has been developed specifically to test gLite User Interface –Certain tests can also be used as functional tests of local site/the whole GRID –Test gLite 3.0 under Scientific Linux CERN 4 –Investigating the problems found Integration of SRM test suites into SAM –Two test suites previously existed for the Storage Resource Manager protocol: LCG-DM tests and s2 srm tests –Needed to extract the desired tests, recompile, etc. –Wrapper scripts for SAM framework are developed

gLite D. Zaborov, SA1-SA3 meeting, Dubna3 Specificity of testing User Interface The aim of User Interface machine is to be an access point to the GRID services –No GRID services are running on a UI –Only client software –Not possible to do a complete test of a UI without employing various GRID services User Interface installations may differ in operating system flavor, default user shell, installation directory, system libraries, etc. –Special effort should be made to ensure integrity of the installation, ability of the software tools to load, correctness of shell environment, etc. No test suite for UI existed before

gLite D. Zaborov, SA1-SA3 meeting, Dubna4 The concept Usually GRID user works in command line  All our tests only use command line. In a sense it is a CLI (command line interface) test suite  C/C++, java, python, … interfaces are to be tested with other, dedicated, test suites  Graphical user tools should be tested separately too Command line tools can be tested with a shell script  Organize the test suite as a collection of shell scripts (easy to write and read the code)‏  It should be possible to run each test separately from the others (without a framework)‏  Test scripts should be easy to run, with minimum number of command line options and no special parameter files Client tools do not change in time  No need to integrate UI tests into an automated testing framework. It is sufficient to re-run the tests after a software update.

gLite D. Zaborov, SA1-SA3 meeting, Dubna5 What to test? Installation integrity Shell environment System time synchronization (ntp)‏ Ability of executables to load (implemented as “version tests”)‏ Basic test of the most “popular” commands –proxy and myproxy commands –job management commands –Information system commands –LCG file catalog commands –LCG data management commands

gLite D. Zaborov, SA1-SA3 meeting, Dubna6 bash? tcsh? All GRID commands should be findable and functional regardless what shell is used  Test the two popular shell flavors: bash and tcsh How to make the same test for two shells?  Write two copies of the test: one in bash and one in tcsh – adopted for the shell environment tests  Write the script in such a way that it can be understood by both shells (“multishell script”) - adopted for the “version” tests  Write tests in the simplest version of bash, “sh” – use the fact that sh does not read any startup files or manipulate shell environment – run the script twice: from tcsh and from bash – solution adopted for many of the tests

gLite D. Zaborov, SA1-SA3 meeting, Dubna7 Shell environment tests System environment: PATH, LD_LIBRARY_PATH, PYTHONPATH, … –Look for non-existing directories gLite-specific environment: GLITE_LOCATION, etc. –Look for undefined variables, non-existing directories

gLite D. Zaborov, SA1-SA3 meeting, Dubna8 Installation integrity All commands mentioned in the gLite User Guide (about 100) should exist All libraries mentioned in the gLite User Guide (currently only a few) should exist A man page should exist for each command

gLite D. Zaborov, SA1-SA3 meeting, Dubna9 Ability to load The commands should return zero exit code, i.e. no error, when it is asked for its version (--version, -version, -V options)‏ –These tests can reveal installation problems such as, for example, python version conflict, absence or version mismatch of a system library, etc.

gLite D. Zaborov, SA1-SA3 meeting, Dubna10 NTP test Amazingly often problems in GRID are caused by a wrong computer clock It is not a waste of time to check the clock This is the only service running on UI machine to be tested

gLite D. Zaborov, SA1-SA3 meeting, Dubna11 Test security-related commands grid-cert-info with various options grid-proxy-info with various options grid-proxy-init/info/destroy chain voms-proxy-info with various options voms-proxy-init/info/destroy chain ensure that voms-proxy-init really uses the file given with the -userconf myproxy-init/info/destroy command chain

gLite D. Zaborov, SA1-SA3 meeting, Dubna12 Test Job Management commands edg-job-list-match edg-submit/get status/get output glite-job-list-match glite-submit/get status/get output glite-wms-job-list-match test glite-wms-submit/get status/get output glite-wms-wait-output.sh - delegate proxy, submit/get status/get output test for the gLite WMS job submission system

gLite D. Zaborov, SA1-SA3 meeting, Dubna13 Test information system commands Get info about CEs using lcg-info Get info about SEs using lcg-info Get info about sites using lcg-infosites Get info from local GIIS using ldapsearch

gLite D. Zaborov, SA1-SA3 meeting, Dubna14 Test LCG File Catalog commands Basic test of lfc-ls Create a directory in LFC, list it and remove Create a directory in LFC, make a symbolic link to it and clean up Create a directory in LFC, set its comment, list, delete comment, delete the directory Create a directory in LFC, list ACL, modify ACL, list ACL, delete directory

gLite D. Zaborov, SA1-SA3 meeting, Dubna15 Test LCG data management commands Create and register, and then remove, a GRID file. Create and register, copy back and remove a GRID file. Create and register, list replica (SURL), get GUID and TURL for the replica, and then delete a GRID file. Create and register, list aliases, create new alias, list again, remove alias, list once again and remove a GRID file.

gLite D. Zaborov, SA1-SA3 meeting, Dubna16 Source code and documentation UI package: org.glite.testsuites.ctb/UI –“Source” code: bin/glite.cgi/org.glite.testsuites.ctb/UI/tests/ TWiki page – #User_Interface_UI

gLite D. Zaborov, SA1-SA3 meeting, Dubna17 Employing the package for real work First the package was used to test new versions of gLite TarUI (tarball version of gLite UI usually installed on computer clusters)‏ Also lxplus installation of gLite UI was tested A number of bugs and other issues have been found using this package Since then the package is used to test new distributions of gLite UI (before releasing)‏ The package may also be helpful for GRID system administrators

gLite D. Zaborov, SA1-SA3 meeting, Dubna18 Problems with gLite 3.0 under SL(C) 4 Python 2.2 vs. 2.3 version conflict (see next slide)‏ –python is a scripting language –Tkinter (a graphics library for python) is also affected Under SL(C)4 certain system libraries (e.g. libssl, libcurl) have different versions than certain gLite components ask for –The solution is to update the gLite components No major issues prevent to port gLite to SLC4 –After a little modification a gLite 3.0 UI compiled for SLC3 could correctly run under SLC4

gLite D. Zaborov, SA1-SA3 meeting, Dubna19 Problems with Python Python 2.2 installed with gLite 3.0 conflicts with python 2.3 installed with SL4 –The problem is due to mixing of two python versions (depends on how $PATH is defined!)‏ –Most python scripts can be interpreted by any of the two versions of python but fail with a mixture –The solution is to remove old python Certain python scripts and modules explicitly ask for python 2.2, mention python 2.2 directories etc. –The solution is to update the scripts A similar conflict was observed with Tkinter –The solution is to update or remove tkinter from gLite distribution

gLite D. Zaborov, SA1-SA3 meeting, Dubna20 Integrating SRM 2.2 test suites into SAM Two test suites for SRM 2.2 protocol have been integrated into the SAM testing framework –LCG-DM SRM 2.2 test suite –S-2 SRM 2.2 test suite

gLite D. Zaborov, SA1-SA3 meeting, Dubna21 What is SRM? The SRM protocol is developed as the main protocol of managing storage resources in GRID –SRM stands for Storage Resource Manager –Each storage resource in the GRID should exhibit an SRM service –SRM protocol defines methods to: Bring a file into SRM’s cache Reserve space before putting a file Release space Request a file transfer between two storage elements Get status of a request etc. –See for more information

gLite D. Zaborov, SA1-SA3 meeting, Dubna22 What is SAM? SAM is a testing and monitoring framework SAM framework consists of two parts: database server and client part –The server part is responsible for collecting and keeping test results and presenting it in a human-readable form (usually on a web-page)‏ –The client part is responsible for collecting exit codes and text output (in html format) of test scripts and publishing the results to the server A test application is compatible with SAM if: –Its exit codes comply with SAM standards –Its text output is compatible with html –It is able to work from current directory (changing directory disrupts writing log-files by SAM)‏ –It comes with a (very short) definition file A SAM application can be written in any programming or scripting language

gLite D. Zaborov, SA1-SA3 meeting, Dubna23 LCG-DM SRM test suite Within gLite the SRM protocol is defined in the LCG Data Management (LCG-DM) package The LCG-DM package has a test suite for each of its components, including SRM Our task was to integrate the test suite for SRM 2.2 into SAM This is done by a wrapper script which –determines the SRM endpoint for an SE host using the information system, –launches the test suite –and puts some html tags into the output stream

gLite D. Zaborov, SA1-SA3 meeting, Dubna24 S-2 SRM test suite There is another test suite for SRM 2.2, written in s-2 programming language –The test suite has been integrated into SAM by means of a wrapper script –The wrapper interrogates the information system to determine the SRM endpoint giving a SE hostname, launches the s-2 tests and collects the results in a single html file

gLite D. Zaborov, SA1-SA3 meeting, Dubna25 Further information See also the work report at te/Report-Jan-Mar-2007.pdf

gLite D. Zaborov, SA1-SA3 meeting, Dubna26 Conclusion A test suite for gLite 3 User Interface has been developed and allows to –test shell environment –test for missing/non-functional commands/libraries –test basic functionality of the most popular commands gLite 3.0 UI has been tested under Scientific Linux CERN 4.4 –The main problem was identified as conflict between python 2.2 and 2.3 –A few minor issues and a number of bugs found Two test suites, dedicated to SRM 2.2, have been integrated into SAM testing framework

gLite D. Zaborov, SA1-SA3 meeting, Dubna27 Acknowledgement This work was supported by CERN-INTAS grant