Discussion on Testing Simantics Technical Board Meeting

Organizing Test Suites Suggestion: new plug-in org.simantics.tests – Contains test suites for different testing activities: Regression TDD Performance Stress – Composes test suites for the SDK from around the codebase, doesn’t contain test cases – Most suites are automatically tested via Jenkins

Testing Activities Regression – First step is to move all currently existing and passing tests into regression tests TDD – Development-time, moved into regression when development is complete, manually executed Performance/Scalability – Focus on performance-critical parts of the SDK right now: databoard, DB, graphics, history Stress – For uncovering hidden unstabilities, mainly relevant for DB

Tools to Incorporate Squish (finally) EMMA (code coverage for unit tests) Possibly static analysis tools

Static Analysis Several (free) tools available for Java [1] that have Eclipse & Jenkins plug-ins :[1] – PMD, FindBugs, Checkstyle AFAIK, none of us has been actively using any of these with Simantics Relatively cheap to automate Need to test the tools first to say anything definitive of usefulness

What to Test? org.simantics.tests: everything that is part of the platform – not products built on top of it (sysdyn, modelica, etc.) A separate test suite for every product UI testing – Difficult without a Workbench product – Use movie tutorial ? – Perhaps create the planned diagram editor for it ?

Reporting Jenkins provides unit/squish test reports Performance testing needs different logging – Instrument all tested code with necessary performance logging (See Eclipse tracing facility)Eclipse tracing facility – See about using Apache JMeter (Jenkins plug-in)Apache JMeterJenkins plug-in For stress tests it is important to have proper logging enabled and relevant data kept for failed tests (databases, etc.)

Epics (prioritized) Create and automate org.simantics.tests Sysdyn product test suite with UI tests List most important testing topics Coverage/Static analysis tools into use Performance testing plan – How to measure? Relativity of results? Reporting/tools? Prevent publishing of builds with broken tests & ensure is delivered about breakage – Try to force developers to react to acute problems

Working with tests and failure Regression tests divided into three suites: – 1st: only intended to contain tests that work. It is important to know when something breaks, and it is easiest to discover this when builds turn from green to red/yellow or vice versa. – 2nd: broken tests that have been moved from 1st if fixing the bug that caused the regression is not possible immediately. – 3rd: broken tests that have no incoming fix for a long time. 2nd->3rd moves done only at the turns of sprints.

References [1]