1. Model-Based Black Box Testing and its Standardization Layers
MaTeLo
THE SOLUTION TO BOOST
YOUR TEST EFFICIENCY

2. EMBEDDED SYSTEMS ALL4TEC Since 1998 French: 45 experts
Turnover 2010: 4 M€
Expertise offer:
PROCESS IMPROVEMENT
SAFETY ENGINEERING
SYSTEM ENGINEERING
OPTIMIZED TESTING
SOFTWARE RELIABILITY
EMBEDDED
SYSTEMS

3. OUR COMPETENCES OFFER Training Consulting Engineering Tooling MaTeLo
EXAM
Safety Architect
M Elopée

4. AGENDA THE TEST PROBLEM THE EXPECTED SOLUTION
THE MODEL-BASED TESTING SOLUTION
A PRAGMATIC INDUSTRIAL USE-CASE
A MORE STANDARDIZED TOOLS INTEGRATION

5. THE TESTING PROBLEM Test Resources Lack Test Focus Deviation Testing
Effort
Time
Test
Needs
System
Complexity
Test
Availability
Usage
Intensity
Product Life
Reliability
Market
Expectation
Customer
Validation
Development
Phase
Reality
Normal
Usage

6. HOW TO SOLVE THIS ISSUE? before product release Test the SUT usage
engineering process
Use best in class
automatic generation
Widely use
Use dedicated tools that decuple engineering productivity
Use formalization
and traceability

7. MODEL-DRIVEN ENGINEERING
Functional
Requirements
DESIGN
TEST
Implement
Scenarize
Design
Model
Usage
Model
MaTeLo
Generate
Generate
Model Based Testing
Code
Test Cases
Compile
Derive
Test
Environment
EXE
Test Scripts

8. WHAT IS MaTeLo? MaTeLo Formal Test Specification More Productive
Product Usage
Point of View
More
Productive
Coverage
Fully
Measured
Process
Systematic
Engineering
Productivity
MaTeLo
Bidirectional
Requirements
Traceability
Markov Test Logic
Test Execution
Compatibility

9. MATELO SCREENSHOT

10. INTEGRATED TEST WORKFLOW
MaTeLo
Testor
Automatic Test
Cases Generation
PLAN
MaTeLo
Usage Model Editor
Test Design into
Usage Models
ACT
MaTeLo
Test Campaign Analysis
Confidence
And Coverage
Reporting
EXAM
Test Automation DO
CHECK

11. MaTeLo EDITOR Usage Model Design

12. MATELO USAGE MODELS Probability on every Alternatives
Association of Requirements
Configuration of Test Operations
Setup Transition as Test Step
BEGINNING
TRANSITION
MACRO CHAIN
STATE
END

13. MODEL TRANSITION = TEST STEP
Test Operations
Select_Gear(5)
Accelerate(100%)
Check_Speed(185)
Requirements
Stimulations
Inputs Stimulation
Equivalence Classes
Timing
Test Oracle
Outputs = f(Inputs)
Verifications
Expected Outputs
Timing

14. Usage path probability
MATELO PROFILE
Profiles can be embedded to qualify the usage model
Operational profiles
Test profiles
Data distribution
Usage path probability
347
87 (+/- 25)

15. MaTeLo EDITOR Requirements Management

16. REQUIREMENTS MANAGEMENT
IBM Rational Doors
MaTeLo Requirements Library
Import
MaTeLo Usage Model
Association
(N,M)
Requirement Update
New Baseline
Quick
Model
Update
Update
Requirements
Change
Analysis
Model
Impact
Highlighting
Available for
other ReqMgt tools
with import of
XML or CSV files

17. MaTeLo TESTOR Test Cases Generation

18. CRITICALITY, COMPLEXITY
MaTeLo TEST STRATEGY
Start
Close
Risk based
approach
Custom
Test profile
CRITICALITY, COMPLEXITY
UPDATE FOCUS
Most probable
approach
Start
Close
FREQUENCY
FOCUS
Arcs coverage
approach
Close
Start
REQUIREMENTS
COVERAGE
Usage
Test profile
approach
Start
Close
OPERATIONAL
COVERAGE
DEFINE THE TEST STRATEGY, BY CHOOSING
Test Algorithm
Test Profile
Part of model to test
Generate

19. MaTeLo TESTOR: HTML TEST PLAN
Input
Expected Result
Requirement
Verdict
State
Test Function

20. Example: MaTeLo FOR EXAM
Association
Test Repository
SUT Interface
Test Operations
Stimulation
Measurement
Administration
Sub Test Sequences
Requirements
Name
UUID
Description
Association
Automatic Test Cases
Automatic call of Test Operation
Automatic parameterisation
Requirements association
Test Case description generation
Usage model respect
Generate

21. MaTeLo TCA Test Campaign Analysis

22. Confidence Indicators
TEST CAMPAIGN PROCESS
New Test
Cases
Generation
Metrics
Quality/Version
Target Quality Criteria
Coverage
Reliability
Confidence Indicators
New
Package
New version
Bugs
Operational Confidence
Needs Coverage
Functional Coverage

23. From Pragmatic Tools Integration To Its Standard Adoption

24. THEORETICAL MBT WORKFLOW
Automation
Test Framework
Test Scheduler
Simulator
Atomic
Test
Action
Confidence
& Coverage
Reports
Test
Scripts
Usage
Model
Test
Cases
Test
Cases
Verdicts
Test
Execution
Log
Functional
Requirements
System
Under
Test

25. A 1ST INDUSTRIAL TOOLS INTEGRATION
EXAM
UML
Python
dSpace
MicroNova
Carts
Vision
CAN
IOs
Doors
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
Test
Scripts
Usage
Model
Test
Cases
Test
Cases
Verdicts
MaTeLo
Simulator
Test
Execution
Log
Functional
Requirements
System
Under
Test

26. A 2nd INDUSTRIAL TOOLS INTEGRATION
TestStand
Proprietary
Opal-RT NI
Clemessy
Diagnostic
Calibration
Radio
Reqtify
MS Excel
MaTeLo
EXAM
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
UML
Test
Scripts
Python
Usage
Model
Test
Cases
Test
Cases
Verdicts
dSpace
MicroNova
Simulator
Carts
Test
Execution
Log
Functional
Requirements
Doors
Vision
CAN
System
Under
Test
IOs

27. VARIOUS TOOLS INTEGRATION
EXAM
TestStand
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
PROVEtech
Proprietary
CANoe
EXACT
TTCN-3 VB
C, C#
CSV
Java
Simulink
Flexray
Temperature
XML
HP QC
Test Scheduler
UML
Test
Scripts
Python
Usage
Model
Test
Cases
Test
Cases
Verdicts
Proprietary
MaTeLo
dSpace NI
Opal-RT
MicroNova
Simulator
Clemessy
Carts
Test
Execution
Log
Functional
Requirements
Doors
Reqtify
Vision
CAN
System
Under
Test
Diagnostic
Calibration
MS Excel
IOs
Radio

28. VARIOUS TOOLS INTEGRATION
TestStand
Proprietary
dSpace
Opal-RT
MicroNova NI
Clemessy
Carts
Vision
Diagnostic
Calibration
CAN
IOs
Radio
Doors
Reqtify
MS Excel
EXAM
UML
Python
PROVEtech
CANoe
EXACT
TTCN-3 VB
C, C#
CSV
Java
Simulink
Flexray
Temperature
XML
HP QC
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
Test
Scripts
Usage
Model
Test
Cases
Test
Cases
Verdicts
MaTeLo
Simulator
Test
Execution
Log
Functional
Requirements
System
Under
Test

29. MBT AND ITS STANDARDS Simulator TestStand Proprietary dSpace Opal-RT
MicroNova NI
Clemessy
Carts
Vision
Diagnostic
Calibration
CAN
IOs
Radio
Doors
Reqtify
MS Excel
EXAM
UML
Python
PROVEtech
CANoe
EXACT
TTCN-3 VB
C, C#
CSV
Java
Simulink
Flexray
Temperature
XML
HP QC
Automation
Test Framework
Atomic
Test
Action
Confidence
& Coverage
Reports
Test Scheduler
ATX
Test
Scripts
ATX
Usage
Model
Test
Cases
Test
Cases
Verdicts
ATX
ATX
OTX
HIL API
MaTeLo
RIF
Simulator
ODS
Test
Execution
Log
Functional
Requirements
FIBEX
XCP
System
Under
Test
ODX
GDI

30. Tools Standard Application Feeling
Items
Tools
Standard
Application
Feeling
Requirements
Doors
RIF, ReqIF
3/5
Test Specification Model
MaTeLo, Word
- (DSL) -
Test Cases Specification
Doors, Word, EXAM
ATX, OTX
0/5 , 2/5
Test Campaign, Management
MaTeLo, Quality Center
ATX
0/5
Test Case Implementation
EXAM, ECU TEST, TestStand, vb, python…
HIL API, TTCN3
1/5 , 2/5
Test Execution
Physical Signal
NI, dSPACE, Vector…
HIL API
1/5
Network Signal
CAN, LIN, FlexRay…
FIBEX
ECU parameter
CANape, INCA
XCP
5/5
Diagnostic
CANdela,
ODX, UDS
Simulator Signal
Simulink, AMEsim…
Test Device Signal
Agilent, LeCroy…
GDI
Test Case Execution Log, Trace
NI, EXAM, dSPACE, ETAS…
ODS, ATML
2/5, 2/5
Test Case Verdict
Quality Center, MaTeLo
Issue
BugZilla, Jira, Quality Center
ISSUE ?
Requirement Coverage
Doors, Quality Center, MaTeLo
ATX, ReqIF
Confidence Indicator
Quality Center, MaTeLo

31. Wiki & Forum & Documentation
QUESTION ?
Sales contact
Wiki & Forum & Documentation

32. ABSTRACTION LAYERS MaTeLo Third Party Equipment
Usage Scenario Description
Markov-Chain Usage Model
MaTeLo
<< derive >>
ATX
Test Case Specification
UML Sequence Diagram
<< generate >>
HIL API
Test Flow Control
Device Driver
Python
Precompiled Application
<< control >>
ODX, FIBEX, XCP…
System Under Test
Hardware In the Loop Test Bench
Third Party
Equipment

33. EXAM ABSTRACTION MANAGEMENT
Possible Usage Scenarios
Test Case 1
Test Case 2
Test Case n
Strategic Test Cases
ATX
HIL Plateform
dSpace
NovaSim
Carts
ASAM HIL API …
Proprietary
Network
CAN_1
CAN_2
LIN_X
FlexRay_1
FlexRay_2
Functions
Manual Gear
Auto Gear
Hand Free
ACC
StartStop
Test Tools
CANoe
CANape
INCA
MS Excel
Diag Tool
Env. Model
Gasoline
Diesel
Turbo
Hybrid
Electric
Available Test Configurations
HIL API
ODX, FIBEX, XCP…
HIL API