1. Challenges and state-of-the-art of neural network verification
Georg Nührenberg

2. fortiss - An-Institut Technische Universität München
Landesforschungsinstitut des Freistaats Bayern
Institute for Software and Systems
Non-profit academic research institute
Challenges and state-of-the-art of neural network verification

3. Application-focused R&D
What we do
Research and Transfer Institute
Apply state-of-the art in industry
Roll-out and transfer of science-based technology
Research on systems & software
Explore research ideas from projects
Collaboration with leading research institutes world-wide
Application-focused R&D
Challenges and state-of-the-art of neural network verification

4. Feed forward neural networks (piecewise linear activations)
Application in intelligent perception and control
Autonomous driving
Control
Perception
lane detection
obstacle classification
input
output
End-to-end (NVIDIA)
hidden layers
Star tracking
spacecraft attitude determination
feed-forward neural network for pattern identification
RELU
𝑦=𝑚𝑎𝑥(0, 𝑤⋅𝑥+𝑏)
Trask, AJ & Coverstone, VL 2003, 'Autonomous artificial neural network star tracker for spacecraft attitude determination' Advances in the Astronautical Sciences, vol 114, no. SUPPL., pp
Challenges and state-of-the-art of neural network verification

5. The need for Neural Network Verification
Dependable neural networks are crucial for safe and secure autonomous and decision systems
Might generate surprising/unpredictable behavior (e.g., adversarial input)
Challenges and state-of-the-art of neural network verification

6. Some recent activities
Resilience of neural networks
Maximal resilience of artificial neural networks (ATVA 2017)
Safety properties of neural networks
Safety of high-way ANN-based motion predictor – verification and certification considerations (DATE 2018)
Special case: Binarized neural networks
Arxiv report
Challenges and state-of-the-art of neural network verification

7. Problem Statement: Resilience
Verification and Resilience Bound of Neural Networks
„Priority lane“ = 98%
Neural Network
How good can your NN resist sensor noise / adversary attack?
A formal, computable, and comparable measure can act as an indicator or as a differentiator
„Priority lane“ = 6%
Neural Network
Challenges and state-of-the-art of neural network verification

8. Maximum Resilience of Neural Networks
We define a resilience metric that can be computed precisely.
E.g., for all input that “strongly“ classifies to “5”, what is the maximum allowed perturbation to still classify as “5”
Challenges and state-of-the-art of neural network verification

9. State-of-the-art in resilience
Variety of methods:
SMT based modeling
Universität Bremen (ATVA 2017)
Oxford (CAV 2017)
MILP based modeling
fortiss (ATVA 2017)
Imperial College London (Arxiv 2017)
enhanced simplex algorithm
ReLuPlex (CAV 2017)
Size of current benchmarks:
~ 300 neurons for local resilience
< 100 neurons for global resilience
Challenge
Achieve applicability for industry
Scalability of the methods needs to be improved
(Future work
Problem: synthetic perturbations can be spurious
introduce noise model
obtain probabilistic robustness measures)
Challenges and state-of-the-art of neural network verification

10. Verifying safety properties of Neural Networks
Safety of highway motion predictor
Properties to be formally verified:
[Example] Is it possible for the controller to suggest go left, while there is already car in the left?
Input:
sensor data
Output:
velocity prediction
Uses the same framework as computing resilience
Specification of properties via convex polyhedra ?
Input set
Output set
Highway motion predictor, being trained under the NGSim dataset
Challenges and state-of-the-art of neural network verification

11. Verification for deep reinforcement learning
Master’s Thesis in collaboration with TUM:
debris collection in the space using deep reinforcement learning
Deep Reinforcement Learning:
underlying is a neural network for policy approximation
Verification goal:
verify safety properties of the learned policy ( = neural network verification)
Challenges and state-of-the-art of neural network verification

12. Verification of Binarized Neural Networks
only binary parameters (instead of floating point)
advantage: very energy-efficient execution (e.g., on embedded devices)
caveat: precision loss
Source: Deep Learning Resources
Verification Method
Reduction to Hardware Verification and SAT solving
Challenge
usability of BNNs in practice
Challenges and state-of-the-art of neural network verification

13. Key Challenges In the scope of neural networks
Methodology: Additional ingredients needed for certifying AI (NN) components, as potentially next-next generation of ECSS standards and ISO 26262
integrate NN-components into safety case
Algorithm & Tools: Symbolic reasoning and tools (verification engine) for verifying properties and deriving characteristics of a neural network
Challenges and state-of-the-art of neural network verification

14. Questions/Challenges for AI components / neural networks
Verification
methods and requirements
Architecture when using AI components
voting
overlaying
Development process
how to test and repair NN
Challenges and state-of-the-art of neural network verification

15. It is hard to specify the requirements
“Current civil aviation certification processes are based
on the idea that the correct behavior of a system must be completely specified and
verified prior to operation.”
“One persistent challenge presented by adaptive systems is the need to define a com-
prehensive set of requirements for the intended behavior. The dynamic nature of
these systems can make it difficult to specify exactly what they will do at run-time.”
[Bhattacharyya, S., Cofer, D., Musliner, D., Mueller, J., & Engstrom, E. (2015, June). Certification considerations for adaptive systems. In Unmanned Aircraft Systems (ICUAS), 2015 International Conference on (pp ). IEEE.]
Challenges and state-of-the-art of neural network verification

16. Questions/Challenges for AI components / neural networks
Verification
methods and requirements
Architecture when using AI components
voting
overlaying
Development process
how to test and repair NN
Challenges and state-of-the-art of neural network verification

17. Architecture when using AI components
Overlaying
Redundancy / Voting
Controller:trained neural network
Certifiable controller
Controller:trained neural network
Voting
Sensor input
Prohibit some actions
Controller:trained neural network
Final output to be used for actuation
Final output to be used for actuation
Output from NN
arbitrary AI controller
Challenges and state-of-the-art of neural network verification

18. Questions/Challenges for AI components / neural networks
Verification
methods and requirements
Architecture when using AI components
voting
overlaying
Development process
how to test and repair NN
Challenges and state-of-the-art of neural network verification

19. Development process Challenges Testing Code review Fixing
neural network
component
input
output
Testing
What is a test?
related to specification challenge
What is test-coverage?
Code review
How to inspect a NN?
Fixing
What is a bug?
failed test, undesired input/output
How to repair?
How to measure / guarantee improvement?
Challenges and state-of-the-art of neural network verification

20. Georg Nührenberg fortiss GmbH Landesforschungsinstitut des Freistaats Bayern An-Institut Technische Universität München Guerickestraße 25 · München · Germany tel
Challenges and state-of-the-art of neural network verification