Quality Assurance Testing Tony Mack PlanetLab Developers Meeting

Overview How do we validate/test our software? o Developers test their own work, submit when they feel its ready. o Does it build? o Developers have different testing methods/processes (unit tests, manual tests, scripts, etc...) o end to end system tests Did we test everything? o Backwards compatible? o Did new development break old functionality?

Objectives Extensible testing system (test harness) o Define a testing process (what are we testing? How do we test it?) o Configurable (Adding/Modifying/Updating test process should be easy) o Flexible (Test harness can run in different environments) o Automated o Developers can integrate their tests into the system Accounting o Test results are logged. Can be reviewed at any time o Track long term debugging efforts/progress

Test Types New development Bug fixes Backwards compatibility o Are we losing required functionality by upgrading/replacing/deprecating o Does current plc software support new node software? o Does new plc software support old node software? Unit tests o Fine grain module testing End to end tests o Do previous operations still work (can we create a slice at plc and log into it on a node) Long term tests o Bugs that arise over time Stress tests o What are the limits? How much of a beating can it take?

Current Test Harness Create a vserver and setup the testing environment (Optional) Download and install the nightly MyPLC (if build failed and no nightly exists, abandon or test in a previously working environment) Configure the PLC and add relevant test data to db o Site, Nodes/People/Slices at site, Slices on nodes, Keys for person accounts, NodeGroups, etc... Run the api uint test(check if all api methods operate as expected and return what expected) Attempt to boot selected nodes (using qemu) o Attempt to boot a 4.2 and 4.1 o If nodes come up in boot bootstate, check if NodeManager is running and creating slices Execute node test scripts (cpu scheduler, planetflow,...)

To Do Long term tests Upgrading a PLC thats already installed Specific NodeManger tests (slice attributes and resource management) Multiple PLC's and federation Hardware diversity found in real nodes