Group Truck Technology, Powetrain Engineering, Control Systems dept. Model Based Testing in a seamless design process model – What is the business case? Group Truck Technology, Powetrain Engineering, Control Systems dept. Sébastien Ruzza

Model Based Testing in a seamless design process model / Objectives In the Automotive industry : The product complexity is increasing every day. More and more requirements are delivered. Verification activities duration are constantly challenged Human resources & test equipment are rationalised Evaluate Model Based Testing using a Control System study – case: What are the benefits and the limitations? What is the business model? What is the process model? Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach, the assumptions Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Requirement driven verification The main system verification approach used at Control Systems is “Requirement driven verification”  Requirements are covered by Test Cases. Specification Verification PXFS – REQ - 001 STC – A PXFS – REQ - 002 PXFS – REQ - 003 STC – B Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations If this approach has lots of advantages, some limitations also exist: If the specification is not exhaustive, the verification will not be exhaustive. The Verification activity is proportional to the number of requirements. In case of Verification time reduction, which risk is taken, if a part of the verification is not executed? Let’s illustrate this… Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Traditional approach: effort is proportional the to number of requirements to cover Satisfaction of the test objective T1 Limitations Set of detected bugs If the verification is stopped at T1, how can we ensure that the potential bugs are not in the set of requirements and associated in use cases we didn’t cover? Testing effort Actual verification duration Required verification duration for a full coverage Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations If this approach has lots of advantages, some limitations also exist: If the specification is not exhaustive, the verification will not be exhaustive. The Verification activity is proportional to the number of requirements. In case of Verification time reduction, which risk is taken, if a part of the verification is not executed? Selection of the test values is tricky. Let’s illustrate this… Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations PXFS- REQ-1: Engine shutdown request shall be sent if Engine Speed < 700 rpm Equivalency Class technique is used to define test cases (define in IEC61508 and ISO15998 and recommended by SPICE) TRUE: Engine shutdown request is sent FALSE: Engine shutdown request is NOT sent Engine Speed 0rpm 700 rpm In practice we select one value in the centre of the range. Is it enough? Often the issue is on ONE singular value. Which one to take? Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Model – Based Testing Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Test Data Distribution Model – Based Testing Why is it a Stochastic approach? S1: Engine is RUNNING Usage model: a representation of how the system will be used under operating conditions. P = 0.05 P = 0.95 S2: Stall Engine S3: Stop & Start shutdown S4: Engine is OFF Test Data Distribution Use path Probability Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

+ + Model – Based Testing 0.9 0.1 What are the input required? Operational profiles & statistics of usage are implemented into the model. They are inputs to: Improve the functional coverage, Execute a Risk driven verification (the most probable) Evaluate the reliability of the system (user oriented) S1: Engine is RUNNING S2: Stall Engine S3: Stop & Start shutdown 0.9 S4: Engine is OFF Specifications FUNCTIONAL REQUIREMENT + 0.1 OPERATIONAL PROFILE + Cope with the non completeness of the specification USE CASE USAGE MODEL Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

VERIFICATION ENGINEER AUTOMATIC TEST SEQUENCES Model – Based Testing What are the output? MANUAL TESTING Different algorithms are used depending on the objective: End – User oriented, reliability, risk driven, … TEST PROCEDURE VERIFICATION ENGINEER Test Generator USAGE MODEL AUTOMATIC TEST SEQUENCES TEST SEQUENCER AUTO TESTING Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

MBT evaluation , Fact & Figures Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results / Process Mapping, Activities Req & design Verification MIL Sys REQ ANALYSIS <test execution> <test execution> HIL System Verif Model Based Testing SYSTEM LEVEL <active review> DESIGN <usage model> SIL <test execution> SW REQ ANALYSIS SIL SW Verif SW LEVEL DESIGN Model Based Design (Simulink/ASCET) Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results / Process Mapping, Role Req & design Verification MIL Sys REQ ANALYSIS HIL System Verif Model Based Testing SYSTEM LEVEL DESIGN Verif Engineer SIL SW REQ ANALYSIS SIL SW Verif SW LEVEL DESIGN Model Based Design (Simulink/ASCET) Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Problem Report Distribution - Type Analysis & Results Some figures Notes: These results correspond to 2 verification phases for 2 different releases. On the graph below, the aim is to show the different types of issues found using MBT. Problem Report Distribution - Type Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Problem Report Distribution – MBT vs. Traditional Analysis & Results Some figures Notes: On the graph below, the aim is to show the number of issues found with the traditional approach and the MBT approach. In parallel to the MBT Verification, manual test was performed on the traditional way. Problem Report Distribution – MBT vs. Traditional Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results Some figures 3 test objectives: User oriented, Structural coverage, Reliability assessment Covered System Requirements: 45 Generated Test procedure: 407 (xml and automatic test sequences) Test Equipment: Hardware In the Loop Auto Test Library: 17 test functions The last results: STATUS Number % Passed 250 61% Failed 157 39% Error 0% TOTAL 407 100%  Driving 19 Problem Reports Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Model – Based Testing / Observation Testing efficiency Better efficiency at the beginning, when the most risky errors are discovered Maximise the testing time on the most probable bugs from a customer prospective Satisfaction of the test objective Traditional approach Model Based Testing Testing effort Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Cost for Test Creation* Cost for Test Execution* Analysis & Results Cost What is the development cost to create the Test Procedures and execute them for 45 Systems Requirements? Model Based Testing Traditional approach For Manual Execution For Auto Execution Test Procedure format Textual test procedure Auto test sequences System Test Cases Max nbre of Test Cases Infinite ~45 Cost for Test Creation* ~10 days ~40 days Cost for Test Execution* ~3 days ~0 (nightly execution) (*): this is an estimation based on a first try and evaluation using one tool from the market. Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Observations & Conclusions Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Keys for success Human resources Technical scope Train the people  This is a different way of testing The generated test procedures for manual execution must not be executed by beginners High level procedures Involve System experts to define the operational profile Secure the correct usage is verified Technical scope Select the appropriate system functions. Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Keys for success Process Test Equipment Map it as a V&V activity in the process Identify clearly who shall deliver: Usage model & Test procedure Test Equipment Development of Automatic steps take more time than expected  Auto test library integration takes time To test the reliability of a system, first, the test equipment must be reliable 407 Auto Test Case stimulate intensively the test equipments Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Question & Answers Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018