1. Trusting Machine Learning Algorithms for Safeguards Applications
Nathan Shoman
SAND C

2. Many different commercial machine learning applications
Images/logos copyright to their respective owners

3. Machine learning can be applied to domains relevant to safeguards
Anomaly detection in multivariate data sets
Zhang (2018)
Anomaly detection in images
Neural Network Prediction
Actual frame
Anomaly
UCSD Dataset

4. Understanding neural networks
Explain forward-pass, back-prop, classification, supervision
Stanford CS231n (2018)

5. More complex networks become difficult to interpret
Inception network
Szegedy et al (2015)

6. Practical considerations for evaluating NN performance
Precision / recall / F1
Importance of validation and test data
Exploring intermediate layers
Layer 5
Layer 3
Zeiler (2013)

7. Using LIME (Local Interpretable Model-agnostic Explanations)
General algorithm to explain predictions of classifiers or regressors by approximating it locally with an interpretable model
Fidelity – Interpretability Trade-off
Riberio, et al. (2016)

8. Using LIME with CNNs for image recognition and classification
Riberio, et al. (2016)

9. One Pixel Attack for Fooling Deep Neural Networks
Su et al. (2017)

10. Conclusions
ML algorithms are powerful tools that could improve existing safeguards and security systems
Trust of machine learning algorithms is essential to acceptance by the safeguards community
Analysis with tools such as LIME is important when presenting results
Newly developed strategies such as Layer-wise Relevance Propagation (Binder, et al. 2016) and Testing with Concept Activation Vectors (Kim et al. 2018) can provide further insight into ML classification logic
One pixel attack is detectable, even when not perceived via human eye, but require extra pre-processing (Xu et al. 2017, Liang et al. 2017)
Binder –
Kim - Xu
Liang