1. CSE 291G : Deep Learning for Sequences
Paper presentation
Topic : Named Entity Recognition
Rithesh

2. Outline Named Entity Recognition and its applications.
Existing methods
Character level feature extraction
RNN : BLSTM- CNNs

3. Named Entity Recognition (NER)

4. Named Entity Recognition (NER)
WHAT ?
Named Entity Recognition (NER)

5. Named Entity Recognition (entity identification, entity chunking & entity extraction)
Locate and classify named entity mentions in unstructured text into predefined categories : person names, organizations, locations, time expressions etc.
Ex : Kim bought 500 shares of IBM in 2010.

6. Named Entity Recognition (entity identification, entity chunking & entity extraction)
Locate and classify named entity mentions in unstructured text into predefined categories : person names, organizations, locations, time expressions etc.
Ex : Kim bought 500 shares of IBM in 2010.

7. Named Entity Recognition (entity identification, entity chunking & entity extraction)
Locate and classify named entity mentions in unstructured text into predefined categories : person names, organizations, locations, time expressions etc.
Ex : Kim bought 500 shares of IBM in 2010.
Person name
Organization
Time

8. Named Entity Recognition (entity identification, entity chunking & entity extraction)
Locate and classify named entity mentions in unstructured text into predefined categories : person names, organizations, locations, time expressions etc.
Ex : Kim bought 500 shares of IBM in 2010.
Importance of locating named entity in a sentence : Ex : Kim bought 500 shares of Bank of America in 2010.
Person name
Organization
Time

9. Named Entity Recognition (NER)
WHAT ?
Named Entity Recognition (NER)
WHY ?

10. Applications of NER Content Recommendations Customer support
Classifying content for news providers
Efficient Searching algorithms QA
Machine Translation Systems
Automatic Summarization system

11. Named Entity Recognition (NER)
WHAT ?
Named Entity Recognition (NER)
WHY ?
HOW ?

12. Approaches :
ML Classification techniques (Ex : SVM, Perceptron model, CRF(Conditional Random Fields)) Drawback : Requires Hand-crafted features
Neural Network Model (By Collobert – Natural Language Processing (almost) from scratch) Drawbacks : (i) Simple Feedforward NN with fixed window size (ii) Depends solely on word embeddings & fails to exploit character level features – prefix, suffix etc.
RNN : LSTM
variable length input and long term memory
First proposed by Hammerton in 2003

13. RNN : LSTM Overcome drawbacks of existing system
Account for variable length input and long term memory
Fails to handle cases in which the ith word of a sentence(S) depends on words at positions greater than ‘i’ in S. Ex : Teddy bears are on sale. Teddy Roosevelt was a great president.

14. RNN : LSTM Overcome drawbacks of existing system
Account for variable length input and long term memory
Fails to handle cases in which the ith word of a sentence(S) depends on words at positions greater than ‘i’ in S. Ex : Teddy bears are on sale. Teddy Roosevelt was a great president.
SOLUTION : Bi-directional LSTM (BLSTM) - Captures Information from the past and from the future.

15. RNN : LSTM Overcome drawbacks of existing system
Account for variable length input and long term memory
Fails to handle cases in which the ith word of a sentence(S) depends on words at positions greater than ‘i’ in S. Ex : Teddy bears are on sale. Teddy Roosevelt was a great president.
SOLUTION : Bi-directional LSTM (BLSTM) - Captures Information from the past and from the future.
Fails to exploit character level features

16. Techniques to capture character level features
Santos and Labeau (2015) proposed a model for character level feature extraction using CNN for NER and POS.
Ling (2015) proposed a model for character level feature extraction using BLSTM for POS.

17. Techniques to capture character level features
Santos and Labeau (2015) proposed a model for character level feature extraction using CNN for NER and POS.
Ling (2015) proposed a model for character level feature extraction using BLSTM for POS.
CNN or BLSTM?

18. Techniques to capture character level features
Santos and Labeau (2015) proposed a model for character level feature extraction using CNN for NER and POS.
Ling (2015) proposed a model for character level feature extraction using BLSTM for POS.
CNN or BLSTM?
BLSTM did not perform significantly better than CNN and also, BLSTM is computationally more expensive to train.

19. Techniques to capture character level features
Santos and Labeau (2015) proposed a model for character level feature extraction using CNN for NER and POS.
Ling (2015) proposed a model for character level feature extraction using BLSTM for POS.
CNN or BLSTM?
BLSTM did not perform significantly better than CNN and also, BLSTM is computationally more expensive to train.
BLSTM : Word level feature extraction
CNN : Character level feature extraction

20. Named Entity Recognition with Bidirectional LSTM-CNNs
Jason P.C. Chiu, Eric Nichols (2016). Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4,
Inspired by :
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural language processing (almost) from scratch. The journal of Machine Learning Research, 12: pages
Cicero Santos, Victor Guimaraes Boosting named entity recognition with neural character embeddings. Proceedings of the fifth Named Entities Workshop,

21. Reference paper : Boosting NER with Neural Character Embeddings
CharWNN deep neural network – uses word and character level representations(embeddings) to perform sequential classification.
HAREM I : Portuguese SPA CoNLL-2002 : Spanish
CharWNN extends Collobert et al.’s (2011) neural network architecture for sequential classification by adding a convolutional layer to extract character-level representations.

22. CharWNN Input : Sentence
Output : For each word in the sentence a score for each class

23. CharWNN Input : Sentence
Output : For each word in the sentence a score for each class
S : <w1, w2, .. wN>

24. CharWNN Input : Sentence
Output : For each word in the sentence a score for each class
S : <w1, w2, .. wN> wn
un =[rwrd; rwch] un

25. CharWNN Input : Sentence
Output : For each word in the sentence a score for each class
S : <w1, w2, .. wN> wn
un =[rwrd; rwch] un

26. CNN for character embedding

27. CNN for character embedding
W : <c1, c2, ..cM>

28. CNN for character embedding
W : <c1, c2, ..cM>

29. CNN for character embedding
W : <c1, c2, ..cM>
Matrix vector operation with window size k

30. CNN for character embedding
W : <c1, c2, ..cM>
Matrix vector operation with window size k

31. CNN for character embedding
W : <c1, c2, ..cM>
Matrix vector operation with window size k
rwch

32. CharWNN Input : Sentence
Output : For each word in the sentence a score for each class
S : <w1, w2, .. wN> wn
un =[rwrd; rwch] un
<u1, u2, .. uN>
rwch

33. CharWNN
Input to convolution layer : <u1, u2, .. uN>

34. Two Neural Network layers
CharWNN
Input to convolution layer : <u1, u2, .. uN>
Two Neural Network layers

35. Two Neural Network layers
CharWNN
Input to convolution layer : <u1, u2, .. uN>
For a Transition score matrix Atu
Two Neural Network layers =

36. Network Training for CharWNN
CharWNN is trained by minimizing the negative log-likelihood over the training set D.
Interpret the sentence score as a conditional probability over a path (the score is exponentiated and normalized with respect to all possible paths)
Stochastic gradient descent (SGD) to minimize the negative log-likelihood with respect to

37. Embeddings
Word level Embedding : For Portuguese NER, the world level embeddings previously trained by Santos, 2004 was used. And for Spanish, Spanish wikipedia was used.
Character level Embedding : Unsupervised learning of character level embeddings was NOT performed. The character level embeddings are initialized by randomly sampling each value from an uniform distribution.

38. Corpus : Portuguese & Spanish

39. Hyperparameters

40. Comparison of different NNs for the SPA CoNLL-2002 corpus

41. Comparison of different NNs for the SPA CoNLL-2002 corpus
Comparison with the state-of-the-art for the SPA
CoNLL-2002 corpus

42. Comparison of different NNs for the HAREM I corpus
Comparison with the State-of-the-art for the HAREM I corpus

43. BLSTM : Word level feature extraction
Chiu, J. P., & Nichols, E. (2016). Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4,
BLSTM : Word level feature extraction
CNN : Character level feature extraction

44. Character Level feature extraction

45. Word level feature extraction

46. Word level feature extraction

47. Embeddings
Word embeddings : 50 dimensional word embeddings released by Collobert (2011b) : Wikipedia & Reuters RCV-I corpus. Also, Stanford’s Glove and Google’s word2vec.
Character embeddings : randomly initialized lookup table with values drawn from a uniform distribution with range [-0.5, 0.5] to output a character embedding of 25 dimensions.

48. Additional Features Additional word level features :
Capitalization feature : allCaps, upperInitial, lowercase, mixedCaps, noinfo.
Lexicons : SENNA and DBpedia

49. Training and Inference
Implementation :
torch7 library
Initial state of LSTM set to zero vectors.
Objective : Maximize sentence level log-likelihood
The objective function and its gradient can be efficiently computed by Dynamic programming.
Viterbi algorithm is used to find the optimal tag sequence [ i ]T that maximizes :
Learning : Training was done by mini-batch stochastic gradient descent (SGD) with a fixed learning rate, and each mini-batch consists of multiple sentences with same number of tokens.

50. Results

51. Results : F1 scores of BLSTM and BLSTM-CNN with various addition features
( emb : Collobert word embeddings, Char : character type feature, caps : capitalization feature, Lex : lexicon feature )

52. Results : Word embeddings

53. Results : Various dropout values

54. Questions to discuss Why BLSTM-CNNs is the best choice?
Is the proposed model Language independent?
Is it a good idea to use additional features( Capitalization, prefix, suffix etc.) ?
Possible Future Works..

55. Thank you!!