1. Ongoing Challenges in Applying V&V Technologies to Automotive Engine Control
Toyota: James Kapinski, Jyotirmoy Deshmukh, Xiaoqing Jin, Hisahiro Ito, Ken Butts
December 11, 2014

2. Toyota MBD Group Our group focus Our group background Our perspective
Toyota Technical Center
Our group focus
Advanced research in V&V for powertrain controller designs
Our group background
Cyber-physical systems (hybrid systems)
Formal verification methods
Our perspective
Focus is on techniques for application-level real-time controller development
Powertrain Control Division
Model-Based Development Group
Verification & Validation

3. Ever-Increasing Complexities of Powertrain Control System
Fuel economy
Safety
Emissions
Driveability
Evolution of engine control:
CPU: 8bit -> 16bit -> 32bit
OS-less -> Real-time OS (MMU-less)
Independent ECU -> Inter-ECU communication (CAN)
Assembly Language  C Language  Block Diagram Language
Need to meet ever-increasing standards -> more complex control code
Engine control code in modern engines can be measured in millions of lines of code!

4. Features of Powertrain Control Software Development
Safety critical
Single core
But multicore is coming!
Hard real-time
Time-triggered tasks
E.g., P+I control, table lookups
Event triggered tasks
E.g., crank angle events
Not much connectivity
Distribution of features across processors not as significant
Performance and functionality critically depends on environment
Exhaustive test is impossible
Continuous variables over unbounded time ⇒ infinite test cases

5. Verification Challenges
Complex models
Large number of states/inputs
Nonlinearities and lots of switching behavior
Variable time delays (delay differential equations)
Look-up-tables
Can contain legacy code or other black-box components
Inconvenient model formats
Many formal tools require format that can be translated into a discrete-state representation or a hybrid automaton
Simulink semantics are closed
Translating formats is time consuming and error prone
Lack of formal requirements
More on this later…

6. Value of Simulation Helps design validation Can uncover bugs
Vital part of control law development
Can uncover bugs
Does not require verification domain knowledge
Engineers are not familiar with
Temporal logic, bounded model checking, theorem provers
Simulations are cheap and usually fast
Test-suites can be shared and built up across models
© The MathWorks

7. Using Simulation for Test and Verification
Let’s use simulation to guide verification and testing approaches
NOT a fundamentally new idea:
Concolic Testing: Sen et al, Kanade et al
Proofs from tests: (Gupta, Rupak, Rybalchenko)
Falsification analysis: (S-TaLiRo: Georgios, Sriram)
Sensitivity-based analysis (Breach: Donzé, Maler)
Coverage-guided simulation (Thao Dang et al)
Sciduction – combining induction and deduction (Seshia, Jha)
…. (please pardon the omissions)
“Sciduction”: Deduction + induction

8. Spectrum of Analysis Techniques
Testing/Control Techniques
Verification
More Scalable
Simulation
Linear Analysis (numerical)
Test Vector Generation for Model Coverage
Concolic Testing
Linear Analysis (symbolic)
(Bounded) Model Checking
Stability
Proofs
Less Scalable
Theorem
Proving
Reachability
Analysis
Less formal/exhaustive
More formal/exhaustive

9. Spectrum of Analysis Techniques
Testing/Control Techniques
Verification
More Scalable
Simulation
Trajectory Splicing
Linear Analysis (numerical)
Coverage-based Testing
Test Vector Generation for Model Coverage
Simulation-Guided Lyapunov/Contraction Analysis
Concolic Testing
Linear Analysis (symbolic)
(Bounded) Model Checking
Stability
Proofs
Less Scalable
Theorem
Proving
Reachability
Analysis
Less formal/exhaustive
More formal/exhaustive

10. Spectrum of Analysis Techniques
Testing/Control Techniques
Verification
Simulation traces to learn contraction metrics for dynamical systems
A. Balkan, J. Deshmukh, J. Kapinski, P. Tabuada. Simulation-guided Contraction Analysis. To appear in the 2015 Indian Control Conference.
Using simulation segments to efficiently search for counterexamples
A. Zutshi, S. Sankaranarayanan, J. Deshmukh, and J. Kapinski. Multiple Shooting, CEGAR-based Falsification for Hybrid Systems. Best Paper
in EMSOFT 2014.
More Scalable
Simulation
Trajectory Splicing
Linear Analysis (numerical)
Coverage-based Testing
Test Vector Generation for Model Coverage
Simulation-Guided Lyapunov/Contraction Analysis
Concolic Testing
Simulation-based testing to maximize coverage of infinite state-space
T. Dreossi, T. Dang, A. Donze, J. Kapinski, X. Jin, J. Deshmukh. Efficient Guiding Strategies for Testing of Temporal Properties of Hybrid Systems. Submitted to the 2015 NASA Formal Methods Symposium.
Linear Analysis (symbolic)
(Bounded) Model Checking
Using simulation traces to learn Lyapunov functions and barrier certificates
Kapinski, J. V. Deshmukh, S. Sankaranarayanan, and N. Aŕechiga. Simulation-guided Lyapunov Analysis for Hybrid Dynamical Systems. In Hybrid Systems: Computation and Control, 2014.
Stability
Proofs
Less Scalable
Theorem
Proving
Reachability
Analysis
Less formal/exhaustive
More formal/exhaustive

11. CPS Requirement Challenges
Implementation ⊨ Requirements ?
Implementation
Implementation
Verification Tool
Requirements
Classic Verification Assumption

12. CPS Requirement Challenges
Results from Integration Tests
Informal
Engineering Insight
Implementation
Implementation
Simulation-based checks
Incomplete
Requirements
The Reality for CPS

13. CPS Requirement Challenges
Requirements are evolving due to CPS-related issues
Environment/software designs evolve concurrently
Not possible to create a plant model that captures all behaviors
Subtle interactions between states/signals are not known before integration test
Definition of correct behaviors exist only in engineer’s brain
Formal requirements are hard for engineers to develop
Existing requirements do not capture all of the desired behaviors
Model may capture appropriate/expected behavior but requirements do not
Could add venn diagram showing behaviors/model/requirements

14. CPS Requirement Challenges
Requirements are evolving due to CPS-related issues
Environment/software designs evolve concurrently
Not possible to create a plant model that captures all behaviors
Subtle interactions between states/signals are not known before integration test
Definition of correct behaviors exist only in engineer’s brain
Formal requirements are hard for engineers to develop
Existing requirements do not capture all of the desired behaviors
Model may capture appropriate/expected behavior but requirements do not
Could add venn diagram showing behaviors/model/requirements
Let’s look at some ideas to address this

15. Requirement Mining†
Sometimes requirements are not in format needed to perform formal verification
Would be useful to automatically obtain formal specifications
Our approach is simulation-based
Seed Traces
Simulink Model
Counter-example Traces
Simulation Traces
Counter-example Found
Obtain tightest parameter for given traces
Falsify requirement using a global optimizer
Candidate Requirement
No Counter-example
Template Requirement
Inferred Requirement
e.g., Overshoot=?, Settling time=?
e.g., Overshoot=5%, Settling time=0.2 sec.
X. Jin, A. Donze, J. V. Deshmukh, and S. A. Seshia. Mining Requirements from Closed-Loop Control Models. In Hybrid Systems: Computation and Control 2013.

16. Learning Requirements
Learning STL requirements from traces
Optimization-guided learning
Enumerate PSTL formulas up to a certain length (i.e., number of nodes in parse tree of formula)
Use Requirement Mining to mine parameter values from traces
Select best feasible formula
Learning Tool
STL requirement
Length is number of predicates and operators
“Best” formula means “most” feasible in relation to given parameter ranges
Traces from Engineer

17. Summary Many V&V challenges for powertrain systems
Due to CPS nature of systems & high complexity
We are encouraged by simulation-guided approaches
Requirements engineering poses significant challenges
Can’t assume we have a thorough set of formal requirements
Let’s consider simulation-guided approaches

18. Thank You! A benchmark powertrain control model described in:
X. Jin, J. Deshmukh, J. Kapinski, K. Ueda, and K. Butts. Powertrain Control Verification Benchmark. In Hybrid Systems: Computation and Control, 2014.
A version of the benchmark model can be found on the Applied Verification for Continuous and Hybrid Systems (ARCH) site:
Paper:
Models: