Model-based testing of complex manufacturing systems: A case study 10th Dutch Testing Day, October 8, 2004 Niels Braspenning Systems Engineering Group, Eindhoven University of Technology

Outline ASML and Tangram Model-based testing (MBT) MBT framework Case: ASML laser subsystem Conclusions Future work Questions

ASML: TWINSCAN Key figures: Supervisory Control (SUN) 10-15 sub-systems (Power PC) Lower level CPUs (firmware) 50 processors 400 sensors, 500 actuators, 12,5 MLOC Language: C (Java, Python, Matlab) Presentation ‘Testing High Technology Developments at ASML’ by Tom Brugman, Dutch Testing Day 2003

Tangram Development activities effort Test time Shipment date

Model-based testing (MBT) Testing is an operational way to check whether a system implementation is correct There is a need for unambiguous specifications and for test automation  MBT: Specification is described in a formal model (unambiguous) Tests can be generated from this formal model (automation) However: Unambiguous specification takes a lot of time (thus modeling also) Current (worldwide) automatic testing covers small and specific test domains

MBT framework documentation, mental models System interface access formal, suitable input for test tool automatic generation and execution of tests unambiguous description of correct system behavior

Case: laser subsystem Functional black box testing of laser subsystem Objectives: Show applicability of MBT with TorX (RU/UT) within ASML Show usability of  (TU/e) specification models for MBT Investigate limitation/shortcomings of the approach communication laser beam

Case: approach E L EP LP Spin LT TorX TDRV Laser Test rack Informal specification E L Specification model () model EP LP Specification model (Promela) translate ASML docs validate/verify Spin verification properties validate/verify Mental model LT convert Test model (Trojka, laser only) Test environment TDRV TorX Test tool Test rack = model = tooling = ASML Laser SUT

Case: informal specification Interface specifications Operational sequences State diagrams (Confidential)

Case: specification model Environment process: Closed system Specific traces for analysis Laser processes: IO: handle communication LS: process commands, execute actions, create responses CLS: hold current laser state Error handling Configurable behavior Initially: Cymer ELS 7600 with laser state behavior only

Case: different models  Simulation only Modeling expressivity (data, time, functions, stochastics, hybrid) Reasonably easy to modify/configure Promela Simulation and verification Less modeling expressivity (workarounds) Less easy to modify/configure Trojka Testing only (open system) Same characteristics as Promela

Case: results 1/2 Validation and verification Testing Mainly good weather (operational) behavior Simulation with  and SPIN Model checking with SPIN Testing Also bad weather (exceptional) behavior Discrepancies are detected by TorX, with different sources: Documentation, model incompleteness, tooling, SUT

Case: results 2/2

Conclusions Proof of concept delivered: Shortcomings/limitations: Applicability of MBT with TorX within ASML Usability of  specification models for MBT Shortcomings/limitations: Manual translation from  to Promela Tested functionality is limited due to limited interface access Problems with data and timing Error injection in SUT is not possible

Future work Laser case: Tangram: Test more functionality Test real laser Tangram: Direct connection between  and TorX Extensions for data, timed and hybrid testing Using simulation models for (early) integration testing Research in test strategy, test infrastructure, and model-based diagnosis

Questions/discussion