1. SSS Validation and Testing September 11, 2003 Rockville, MD William McLendon Neil Pundit Erik DeBenedictis Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000.

2. APItest Release Testing Experiences at Sandia Status daemon Overview

3. Distributed Runtime System Testing Complex system of interactions Approach to testing –Component Testing –Benchmarks Performance / Functionality –Operational Profile –Stress Testing Users expect a high-degree of quality in today’s high end systems!

4. APItest

5. APITEST - Overview Unit-testing tool for network components –Targeted for networked applications –Extensible framework –Dependency calculus for inter-test relationships Scriptable Tests (XML Schema Grammar) Multi-Protocol Support –TCP/IP, SSSLib, Portals, HTTP

6. Accomplishments Since Last Meeting Spent a week at Argonne (July) –Major rework of framework of APItest Individual tests are atomic. Framework handles the hard work of checking tests, dependencies, and aggregate results. –Extensibility New test types are easy to create Dependency System –Define relationships as a DAG encoded in XML. –Boolean dependencies on edges.

7. Supported Test Types sssTest –use ssslib to communicate with ssslib enabled components shellTest –execute a command httpTest –ie. app testing web interfaces (a’la Globus, etc) tcpipTest –raw socket via tcpip transmission.

8. APITEST supports Regular Expressions Regular expression matching in tests Sent to service-directory: Expected Buffer: Regular Expression: (\s* \s*)+

9. Creating New Test Types is Easy A simple test that will always pass: class passTest(Test): __attrfields__=[‘name’] typemap = {‘dependencies’:TODependencies} def setup(self): pass def execute(self, scratch): self.expect[‘foo’] = [ (‘REGEXP’, ‘a’) ] self.response[‘foo’] = ‘a’

10. Matching and Aggregation An individual test can be executed many times in a sequence. –PASS/FAIL can be determined based on the percent of runs that matched. –Percent Match can be specified as a range as well. Expected result is specified as a regular expression (REGEX) or a string for exact matching (TXTSTR) Notation: –M [min:max] - Percent matching. min/max = bounds on % of tests where actual and expected results match. If the actual number of tests is within the range specified the test will PASS, otherwise it will FAIL.

11. Test Dependencies T iff A [40:90] OR B [0:0] M [0:0] AB ? M [40:90]  AB M [100:]  C  ? M [90:] M [:] T iff (A [100:]  B [90:] )  C [:] M [40:90] : >= 40% and <= 90% of test runs matched

12. An Example Dependency ( A [100:]  ( (B [100:100]  C)  D [:0] ) )

13. An Example Test Sequence reset daemon test daemon test other stuff M [:] M [:30]

14. Standard Test Metadata Attributes AttributeTypeReq’dDefaultDesc namestringYESname of test numRepsintegerNO1number of times to execute test minPctMatchfloatNO0.0min % of repetitions that must match for test to pass maxPctMatchfloatNO100.0max % of repetitions that must match for test to pass preDelayfloatNO0.0delay in seconds prior to executing test postDelayfloatNO0.0delay in seconds after executing test before continuing to next test. iterDelayfloatNO0.0delay in seconds between tests during loop (only used if numReps > 1) onMismatchstringNOCONTINUEIf a test fails to match, what do we do? CONTINUE,BREAK,HALT

15. Example Scripts A simple shell execution test: Test with a dependency and stdout matching:.*stdout.*

16. APItest Output iterationstest name% matchedPass/Failmessage ----------------------------------------------- [1 of 1]A 100.00%PASS [1 of 1]K 100.00%FAIL m[0.0% : 0.0%] [1 of 1]J 0.00%FAIL m[90.0% : 90.0%] [5 of 5]M 100.00%PASS [1 of 1]L 100.00%FAIL m[0.0% : 0.0%] [1 of 1]N 100.00%PASS [0 of 1]T DEPENDENCY FAILURE(S) F expected [0.0% : 90.0%], got 100.0 J expected [90.0% : 100.0%], got 0.0 [0 of 1]R DEPENDENCY FAILURE(S) J expected [90.0% : 100.0%], got 0.0 K expected [0.0% : 90.0%], got 100.0 [1 of 1]S 100.00%PASS [1 of 1]U1 100.00%PASS [1 of 1]U2 100.00%PASS [0 of 1]U3 DEPENDENCY FAILURE(S) N expected [0.0% : 90.0%], got 100.0 S expected [0.0% : 90.0%], got 100.0 [1 of 1]U4 100.00%PASS

17. sssTest outputs from Chiba City iterations test name% matched Pass/Fail message ---------- ------------------ --------- ---------- [1 of 1] add-location100.00% PASS [1 of 1] QuerySDComps100.00% PASS [1 of 1] QuerySDHost100.00% PASS [1 of 1] QuerySDProtocol 100.00% PASS [1 of 1] QuerySDPort100.00% PASS [1 of 1] del-location100.00% PASS [1 of 1] val-removal100.00% PASS iterations test name % matched Pass/Fail message ---------- --------- --------- --------- ---------- [1 of 1] sss-getproto 100.00% PASS [1 of 1] sss-getport 100.00% PASS [1 of 1] sss-gethost 100.00% PASS [1 of 1] sss-getcomp 100.00% PASS [1 of 1] sss-getproto 100.00% PASS [1 of 1] sss-getport 100.00% PASS [1 of 1] sss-gethost 100.00% PASS [1 of 1] sss-getcomp 100.00% PASS

18. Release Testing…

19. Tales from Cplant Release Testing Methodical execution of production jobs and 3rd Party benchmarks to identify system instabilities, enabling them to be resolved. Ie: –Rapid job turnover rate (caused mismatches between scheduler and allocator) –Heavy io (I/O which passes through launch node process instead of directly to ENFS “yod-io”) Wrapping above codes into Ctest framework to enable portable compile, launch, and analysis of synthetic workloads

20. Ctest Extension of Mike Carifio’s work –Presented at the SciDAC meeting in Houston during Fall of 2002 –Make structure that holds a suite of independent applications. –Tools to launch as a reproducible workload. –Goal: 30 users and 60 concurrent apps

21. Sample Load Profile on CPlant

22. Issue Tracking SNL uses a program called RT –Centralized repository for issue tracking helps give an overall picture of what problems are. –Helps give summary of progress. Bugzilla is on the SciDAC SSS website –http://bugzilla.mcs.anl.gov/scidac-sss/http://bugzilla.mcs.anl.gov/scidac-sss/ –Who’s using it?

23. Status Daemon …

24. Status Daemon Highly configurable monitoring infrastructure for clusters. –Does not need to run daemon on the node you are monitoring. –XML configurable –Web interface “Cluster Aware” Used on CPlant production clusters James Laros (jhlaros@sandia.gov)

25. Status Daemon Communication Admin Node Local Tests Daemon Disk Status Update Status XML Config File XML Data Remote Tests Remote Tests Compute nodes Leader Local Tests Daemon XML Data Remote Tests Compute nodes Leader Local Tests Daemon XML Data

32. Summary New Hire –Ron Oldfield APItest functionality and flexibility increases Release testing experience Status Daemon

33. Plans APItest –User / Programmer Manuals –User Interface GUI? HTTP? –daemon mode for parallel testing mode –DB Connectivity –Test Development ssslib event tests HTTPtest work ptlTest (SNL) SWP Integration –Port SWP to Chiba for SC2003?

34. Questions?

35. END OF PRESENTATION

36. APITEST Component Architecture C bot Tests DB (MySQL) report generator APItest backup/ archival

37. Encoding XML into XML CDATA test1 sd <get-location> <location component='*'/> </get-location> XML parsers will interpret the “<“ character as a new Element, NOT as a character buffer. So,we must trick our SAX parse by using < instead of <