Group Truck Technology, Powetrain Engineering, Control Systems dept.

Slides:



Advertisements
Similar presentations
Formal Methods and Testing Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development.
Advertisements

Configuration management
DETAILED DESIGN, IMPLEMENTATIONA AND TESTING Instructor: Dr. Hany H. Ammar Dept. of Computer Science and Electrical Engineering, WVU.
Software Process Models
MSIS 110: Introduction to Computers; Instructor: S. Mathiyalakan1 Systems Design, Implementation, Maintenance, and Review Chapter 13.
Introduction to Software Testing
Chapter 13 & 14 Software Testing Strategies and Techniques
Software Testing Verification and validation planning Software inspections Software Inspection vs. Testing Automated static analysis Cleanroom software.
System/Software Testing
An Introduction to MBT  what, why and when 张 坚
 To explain the importance of software configuration management (CM)  To describe key CM activities namely CM planning, change management, version management.
Topic (1)Software Engineering (601321)1 Introduction Complex and large SW. SW crises Expensive HW. Custom SW. Batch execution.
Principles of Information Systems, Sixth Edition Systems Design, Implementation, Maintenance, and Review Chapter 13.
B. Fernández, D. Darvas, E. Blanco Formal methods appliedto PLC code verification Automation seminar CERN – IFAC (CEA) 02/06/2014.
Unit Testing 101 Black Box v. White Box. Definition of V&V Verification - is the product correct Validation - is it the correct product.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 22 Slide 1 Software Verification, Validation and Testing.
Topics Covered Phase 1: Preliminary investigation Phase 1: Preliminary investigation Phase 2: Feasibility Study Phase 2: Feasibility Study Phase 3: System.
Verification and Validation Assuring that a software system meets a user's needs.
Chapter 8 Lecture 1 Software Testing. Program testing Testing is intended to show that a program does what it is intended to do and to discover program.
Principles of Information Systems, Sixth Edition 1 Systems Design, Implementation, Maintenance, and Review Chapter 13.
Software Quality Assurance and Testing Fazal Rehman Shamil.
C++ for Engineers and Scientists, Second Edition 1 Problem Solution and Software Development Software development procedure: method for solving problems.
Verification vs. Validation Verification: "Are we building the product right?" The software should conform to its specification.The software should conform.
RANDOM TESTING & PROTOTYPING Presented By: Presented By: Vipul Rastogi M.Tech (S.E.) SRMSCET Presented To: Dr. Himanshu Hora Chief Proctor SRMSCET.
Project and Project Formulation and Management
Information Systems Development
Testability.
Agile Metrics that Matter
INFORMATION SYSTEMS SECURITY AND CONTROL.
PREPARED BY G.VIJAYA KUMAR ASST.PROFESSOR
Information Systems Development
Introduction Edited by Enas Naffar using the following textbooks: - A concise introduction to Software Engineering - Software Engineering for students-
Software Engineering (CSI 321)
Building Information Systems
Software Quality Engineering
A scalable approach for Test Automation in Vector CAST/Manage with
Testing and Debugging PPT By :Dr. R. Mall.
CSC 480 Software Engineering
INCOSE Usability Working Group
SOFTWARE TESTING OVERVIEW
Software Quality Engineering
Chapter 8 – Software Testing
The Systems Engineering Context
Introduction SOFTWARE ENGINEERING.
TechStambha PMP Certification Training
^ About the.
CSE 403 Software Engineering
Building Information Systems
DEFECT PREDICTION : USING MACHINE LEARNING
Chapter 13 & 14 Software Testing Strategies and Techniques
IS442 Information Systems Engineering
Auditing Application Controls
Introduction Edited by Enas Naffar using the following textbooks: - A concise introduction to Software Engineering - Software Engineering for students-
BU IS GIG Chemical, Oil & Gas
Information Systems Development
Engineering Processes
Introduction to Software Testing
Lecture 09:Software Testing
MBML_Efficient Testing Methodology for Machine Learning
A test technique is a recipe these tasks that will reveal something
Model Checking for an Executable Subset of UML
Fundamental Test Process
Introducing ISTQB Agile Foundation Extending the ISTQB Program’s Support Further Presented by Rex Black, CTAL Copyright © 2014 ASTQB 1.
CS240: Advanced Programming Concepts
Welcome to Corporate Training -1
Test Case Test case Describes an input Description and an expected output Description. Test case ID Section 1: Before execution Section 2: After execution.
Software Verification, Validation, and Acceptance Testing
KNOWLEDGE MANAGEMENT (KM) Session # 36
GSBPM AND ISO AS QUALITY MANAGEMENT SYSTEM TOOLS: AZERBAIJAN EXPERIENCE Yusif Yusifov, Deputy Chairman of the State Statistical Committee of the Republic.
Chapter 13 & 14 Software Testing Strategies and Techniques 1 Software Engineering: A Practitioner’s Approach, 6th edition by Roger S. Pressman.
Presentation transcript:

Group Truck Technology, Powetrain Engineering, Control Systems dept. Model Based Testing in a seamless design process model – What is the business case? Group Truck Technology, Powetrain Engineering, Control Systems dept. Sébastien Ruzza

Model Based Testing in a seamless design process model / Objectives In the Automotive industry : The product complexity is increasing every day. More and more requirements are delivered. Verification activities duration are constantly challenged Human resources & test equipment are rationalised Evaluate Model Based Testing using a Control System study – case: What are the benefits and the limitations? What is the business model? What is the process model? Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach, the assumptions Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Requirement driven verification The main system verification approach used at Control Systems is “Requirement driven verification”  Requirements are covered by Test Cases. Specification Verification PXFS – REQ - 001 STC – A PXFS – REQ - 002 PXFS – REQ - 003 STC – B Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations If this approach has lots of advantages, some limitations also exist: If the specification is not exhaustive, the verification will not be exhaustive. The Verification activity is proportional to the number of requirements. In case of Verification time reduction, which risk is taken, if a part of the verification is not executed? Let’s illustrate this… Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Traditional approach: effort is proportional the to number of requirements to cover Satisfaction of the test objective T1 Limitations Set of detected bugs If the verification is stopped at T1, how can we ensure that the potential bugs are not in the set of requirements and associated in use cases we didn’t cover? Testing effort Actual verification duration Required verification duration for a full coverage Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations If this approach has lots of advantages, some limitations also exist: If the specification is not exhaustive, the verification will not be exhaustive. The Verification activity is proportional to the number of requirements. In case of Verification time reduction, which risk is taken, if a part of the verification is not executed? Selection of the test values is tricky. Let’s illustrate this… Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Traditional Verification approach Limitations PXFS- REQ-1: Engine shutdown request shall be sent if Engine Speed < 700 rpm Equivalency Class technique is used to define test cases (define in IEC61508 and ISO15998 and recommended by SPICE) TRUE: Engine shutdown request is sent FALSE: Engine shutdown request is NOT sent Engine Speed 0rpm 700 rpm In practice we select one value in the centre of the range. Is it enough? Often the issue is on ONE singular value. Which one to take? Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Model – Based Testing Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Test Data Distribution Model – Based Testing Why is it a Stochastic approach? S1: Engine is RUNNING Usage model: a representation of how the system will be used under operating conditions. P = 0.05 P = 0.95 S2: Stall Engine S3: Stop & Start shutdown S4: Engine is OFF Test Data Distribution Use path Probability Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

+ + Model – Based Testing 0.9 0.1 What are the input required? Operational profiles & statistics of usage are implemented into the model. They are inputs to: Improve the functional coverage, Execute a Risk driven verification (the most probable) Evaluate the reliability of the system (user oriented) S1: Engine is RUNNING S2: Stall Engine S3: Stop & Start shutdown 0.9 S4: Engine is OFF Specifications FUNCTIONAL REQUIREMENT + 0.1 OPERATIONAL PROFILE + Cope with the non completeness of the specification USE CASE USAGE MODEL Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

VERIFICATION ENGINEER AUTOMATIC TEST SEQUENCES Model – Based Testing What are the output? MANUAL TESTING Different algorithms are used depending on the objective: End – User oriented, reliability, risk driven, … TEST PROCEDURE VERIFICATION ENGINEER Test Generator USAGE MODEL AUTOMATIC TEST SEQUENCES TEST SEQUENCER AUTO TESTING Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

MBT evaluation , Fact & Figures Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results / Process Mapping, Activities Req & design Verification MIL Sys REQ ANALYSIS <test execution> <test execution> HIL System Verif Model Based Testing SYSTEM LEVEL <active review> DESIGN <usage model> SIL <test execution> SW REQ ANALYSIS SIL SW Verif SW LEVEL DESIGN Model Based Design (Simulink/ASCET) Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results / Process Mapping, Role Req & design Verification MIL Sys REQ ANALYSIS HIL System Verif Model Based Testing SYSTEM LEVEL DESIGN Verif Engineer SIL SW REQ ANALYSIS SIL SW Verif SW LEVEL DESIGN Model Based Design (Simulink/ASCET) Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Problem Report Distribution - Type Analysis & Results Some figures Notes: These results correspond to 2 verification phases for 2 different releases. On the graph below, the aim is to show the different types of issues found using MBT. Problem Report Distribution - Type Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Problem Report Distribution – MBT vs. Traditional Analysis & Results Some figures Notes: On the graph below, the aim is to show the number of issues found with the traditional approach and the MBT approach. In parallel to the MBT Verification, manual test was performed on the traditional way. Problem Report Distribution – MBT vs. Traditional Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Analysis & Results Some figures 3 test objectives: User oriented, Structural coverage, Reliability assessment Covered System Requirements: 45 Generated Test procedure: 407 (xml and automatic test sequences) Test Equipment: Hardware In the Loop Auto Test Library: 17 test functions The last results: STATUS Number % Passed 250 61% Failed 157 39% Error 0% TOTAL 407 100%  Driving 19 Problem Reports Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Model – Based Testing / Observation Testing efficiency Better efficiency at the beginning, when the most risky errors are discovered Maximise the testing time on the most probable bugs from a customer prospective Satisfaction of the test objective Traditional approach Model Based Testing Testing effort Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Cost for Test Creation* Cost for Test Execution* Analysis & Results Cost What is the development cost to create the Test Procedures and execute them for 45 Systems Requirements? Model Based Testing Traditional approach For Manual Execution For Auto Execution Test Procedure format Textual test procedure Auto test sequences System Test Cases Max nbre of Test Cases Infinite ~45 Cost for Test Creation* ~10 days ~40 days Cost for Test Execution* ~3 days ~0 (nightly execution) (*): this is an estimation based on a first try and evaluation using one tool from the market. Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Observations & Conclusions Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Keys for success Human resources Technical scope Train the people  This is a different way of testing The generated test procedures for manual execution must not be executed by beginners High level procedures Involve System experts to define the operational profile Secure the correct usage is verified Technical scope Select the appropriate system functions. Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Keys for success Process Test Equipment Map it as a V&V activity in the process Identify clearly who shall deliver: Usage model & Test procedure Test Equipment Development of Automatic steps take more time than expected  Auto test library integration takes time To test the reliability of a system, first, the test equipment must be reliable 407 Auto Test Case stimulate intensively the test equipments Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018

Question & Answers Group Truck Technology, Powertrain Engineering, Sébastien Ruzza 12/5/2018