1. Attention is not Explanation
NAACL’19
Sarthak Jain, Byron C. Wallace
Northeastern University
/647,31%, /1198,22.6%
5-5-5, 5-5-3

2. Background
Attention Mechanism

3. Background-Attention
Given sequence h and query Q
Calculate attention distribution
Additive function
Scaled dot-product function
Get attention vector:

4. Question
Is the attention mechanism really get the semantic attention?

5. Is the attention provide transparency?
Do attention weights correlate with measures of feature importance?
Would alternative attention weights necessarily yield different predictions?

6. Experiment Model y h dense layer encoder (BiRNN) attention h embedding
one hot h Q

7. Dataset

8. Correlation with Feature Importance
Gradient based measure
Leave one feature out

9. Result for Correlation
Orange=>Positive, Purple=>Negative O,P,G=>Neutral, Contradiction, Entailment
Gradients

10. Result for Correlation
Leave One Out

11. Statistically Significant

12. Random Attention Weights

13. Result for Random Permutation
Orange=>Positive, Purple=>Negative
O,P,G=>Neutral, Contradiction, Entailment

14. Adversarial Attention
Optimize a relaxed version with Adam SGD

15. Result for Adversarial Attention
0.69

16. Conclusion
correlation between feature importance measures and learned attention weights is weak
counterfactual attentions often have no effect on model output
limitations
only consider a handful of attention variants
only evaluate tasks with unstructured output spaces (no seq2seq)

17. Adversarial Heatmaps Example

18. Adversarial Heatmaps Example