1. Development of test suites for the certification of EGEE-II Grid middleware
Task 2:
The development of testing procedures focused on special details of various software features
Task 4:
Creating the specialized testbed for developing test suites
Task 5:
Preparing intermediate and final reports
PNPI – Yu. Ryabov, N. Klopov

2. Plans for the second year
Development of the stress and performance tests for WMS and CE according with requests from developers and/or certification team
2. New gLite 3.1 middleware installation on the testbed

3. Requirements to the test
Submit a large number of jobs simultaneously
Submit jobs from one or many users.
Monitoring of a load of CE and WMS during testing.
Monitoring of jobs status (pass through system’s components) on the CE and WMS during testing.
Storing of status information for all submitted jobs.
Possibility of express visual analysis of results.

4. Functional schema of the test
Monitoring
Monitoring CE
WMS
Monitoring data
Jobs logging info
Jobs logging info UI
…..
Data
collector
Job submitter zapuskaet v backgrounde neskol’ko scriptov, kagdii iz kotorih zapuskaet I monitoriruet parametric job ot imeni konkretnogo usera.
Na WMS and CE pered testirovaniem zapuskaetsya monitor, kotorii sledit za load average I zagrugennimi processami. Posle zaversheniya testa Job collector zabiraet dannie monitorirovaniya s CE I WMS, a takge zaprashivaet po komande glite-wms-job-logging-info statusnuyu informaciyu dlya vseh subjobs vsex parametric jobs. Eta informaciya m.b. peredana dlya express visual analysis na web site.
P.S. V kachestve programmi, kotoraya zapuskaetsya na work nodaes, used simple bash script which sleep zadannoe chislo secund
(esli sprosyat)
Parametric job
Parametric job
Job submitter

5. Jobs submission
Job submission program (several scripts) has the
following input parameters:
u- the number of the users
x- path to the directory with users proxy certificates (x1- path to the user proxy certificate)
n -the number of the subjobs from each user
s- time interval between jobs status request
t -max time of the test execution
a- the time of a subjob will execute on WN
l- path to the logfile

6. These scripts run on CE and WMS and provide
Monitoring
These scripts run on CE and WMS and provide
receiving and saving information about load
average and system processes names.
The script runs with the following parameters:
t - pool time
l -request for load average
p -request for process names
Load average ~The quantity of active processes (from UNIX)

7. copy monitoring data from WMS and CE;
Data collector
The Data collector script is executed after finish of all jobs and does the following:
copy monitoring data from WMS and CE;
request the event time information for each subjob, using glite-wms-job-logging-info command;
preliminary data processing (formatting);
Data processing – formatirovanie dannix chtobi bilo udobno obrabativat’ na saite v cgi scriptah

8. Parametric job
Parametric job functionality was used to solve the problem of simultaneous submission of large number of jobs to CE.
Parametric job is a set of jobs (subjobs) with the same descriptions apart from the values of the parametric attributes.
JobType = "Parametric";
Executable = "tst.sh";
InputSandbox = {“tst.sh",
"input_PARAM_.txt"};
StdOutput = "out_PARAM_.txt";
StdError="err_PARAM_.txt";
OutputSandbox={"out_PARAM_.txt",
"err_PARAM_.txt"};
Parameters=1000;
ParameterStart=0;
ParameterStep=1;
Parametric attributes get values from 0 to 999.
WMS will create individual subjob for each parameter value.
N=(Parameters-ParameterStart)/ParameterStep
subjobs will be created
Both main parametric job and its subjobs will
have unique IDs.

9. Testbed WN WMS+LB+ BDII CE WN WN UI WN
gLite 3.1 middleware was installed on the testbed: WN
WMS+LB+ BDII CE WN WN UI WN

10. Test usage
Measurement of “load average” as function of time under the following condition: N jobs from each of K users
Test usage in PNPI:
1000 jobs from each user (1 user, 3 users) for “old” and “new” versions gLite;
Old - we had been using till January 2008
New (with marshal patches) - we have been using since January 2008
New version with marshal patches was released to production 10 April 2008 (gLite update 23)
Marshal patches was developed by A.Kiryanov (PNPI)

11. Marshal patches for LCG CE
Aim is to improve behavior of LCG CEs under load by regulating requests from job managers (hence the term ‘marshal’) due to :
Eliminate the necessity to recompile heavy Perl code on every job manager invocation
Memory-persistent daemons handle the requests
Control of the number of simultaneously running job manager queries
Decreases load on file system and batch system
Prevent CE overload by WMSes
Decrease system’s losses
Jobs complete faster, especially visible with large number of short jobs

12. CE monitoring
Dlya 1 usera net suschestvennih izmenenii dlya staroi I new version

13. CE monitoring
Dlya 3 users (kagdii po 1000 subjobs) – zametnaya raznica po load average and time vipolneniya vseh subjobs

14. WMS monitoring
Zagruzka WMS malen’kaya v oboih sluchayah

15. Express visual analysis (WEB viewer)
Each job passes through the different WMS components (the corresponding events are generated and stored in LB. Example of these events: “RegJob,NetworkServer”,
“Match,WorkloadManager”,…,”Done, LogMonitor”).
It gives the possibility to evaluate the performance of the WMS components.
The WEB viewer was developed to provide the visual representation of
events timestamp for the jobs running through the different components.
This viewer provide the following functions:
- to choose the event type which will be sorted by the timestamp value;
to choose data file with logging info data;
to get the graph of the event time since job registration in WMS for each job;
to choose the additional event type (will be represented on the same graph);
- to get and store graph data as text file for the future analysis;
- to get ID and logging info data for the subjobs those lost the chosen events;
- to view monitoring data.
Data processing – formatirovanie dannix chtobi bilo udobno obrabativat’ na saite v cgi scriptah

16. Express visual analysis
Transfer
(source- Logmonitor
destination- LRMS)
Sostav saita – html stranici and cgi-scripts
Na picture vidno kak event Running ot Logmonitor otstaet ot event Running
P.S. slaid s animaciei
Accepted
(source Logmonitor)

17. Express visual analysis
We can view the monitoring data
Est’ vozmognost’ rassmatrivat’ dannie monitorirovaniya CE and WMS.
Link:
Posle etogo slaida nugen slaid pro novuyu versiyu

18. Summary The testbed was created with the gLite 3.1
A complex test was developed which provide the following:
Submission of the large number of jobs from many users
Load average monitoring on WMS and CE
Data acquisition of the test results
Developed test has been used on concrete sets of input parameters
HTML viewer was developed for the presentation of test results

19. Summary (First year of the grant)
Set of WMS tests (control of functionality) was
developed according to the request from gLite
certification team for the following types of jobs:
parametric, interactive, checkpointable, partitionable.
Long and complex JDL stress test
(for estimation of critical size of file)
Some of the tests were included into certification
SAM framework.
5 bugs were found and submitted in Savannah.

20. Conclusion Task 2 (PNPI)- done Task 4 (PNPI)-done
Task 5 –under preparation
(together with collaborating teams)