Bidirectional CRF for NER Jin Mao Postdoc, School of Information, University of Arizona Sept 14th, 2016

CRF Interpretation Sequence Orders Feature Styles Model Integration AGENDA CRF Interpretation Sequence Orders Feature Styles Model Integration

CRF Interpretation CRF Let X :=(x1,...,xT ) be a sequence of tokens from a tokenized input sentence, Y :=(y1,...,yT ) be a sequence of tags. A CRF tagger selects Y that maximizes its conditional probability given X: a vector of binary features weights

CRF Interpretation CRF The probability of a tag yt depends only on its neighboring tags, given the entire sentence X The tagging by the CRF models is mainly determined by the features fα(yt−1,yt ,X) and their corresponding weights θα.

Define features in factored representation (Sha and Pereira, 2003) CRF Interpretation Feature function Define features in factored representation (Sha and Pereira, 2003) p(X,t) is a predicate (i.e. a boolean function) on X and current position t q(yt−1,yt ) is a predicate on pairs of tags, transition predicate

CRF Sequence Orders Forward VS Backward Forward Backward T-1+1 T-t+2

CRF Sequence Orders Forward VS Backward For t-th token, used the Kullback-Leibler(KL) divergence to measure the information gain from a prior distribution of yt to its posterior distribution after either previous or next tag becomes available: The training corpus from BioCreative 2: The next tag could be more helpful than the previous tag

CRF Sequence Orders Forward VS Backward The prediction accuracy of tag bigrams

HMM-style features described in Lafferty et al. (2001) CRF Feature Styles Feature function MALLET style: HMM-style features described in Lafferty et al. (2001) observation-independent state transition features

A CRF model is symmetric. CRF Feature Styles Symmetric Property A CRF model is symmetric.

CRF Feature Styles CRF with HMM-style features is symmetric, if: Symmetric Property CRF with HMM-style features is symmetric, if: two special tags are attached to the head and tail of Y the training set is the same with different orders all p predicates are either defined on a single token or symmetrically with regard to current position t.

CRF Feature Styles Thus, Symmetric Property Forward T Backward T-t+2 g2(yB; T-(t-1)+1) But, g1 + g2 for t-th token are not identical?

CRF Feature Styles Mallet-style: Symmetric Property t Forward Backward T-t+2 T-t+1 T-t T-t+1 T-1+1 t-1 Mallet-style: g1 is not symmetric any more! g1(X,Y; t) ≠ g1(XB,YB; tB)

CRF Feature Styles CRF++: HMM-style (default), Mallet-style(optional) Tools CRF++: HMM-style (default), Mallet-style(optional) Mallet: Mallet-style(default), HMM-style and others(optional)

CRF Model Integration Label results 10 best results with scores Find matches

CRF Model Integration Integration Method Simple set operations, intersection and union, failed to improve the performance because they lead to trade-off between recall and precision. Heuristic method: (1) compute the intersection of bi-directional parsing and select the solution in the intersection that minimizes the sum of its output scores; (2) for the other 18 solutions, select the labeled terms appearing in a dictionary with its length greater than three. approved gene symbols and aliases obtained from HUGO (Eyre et al., 2006) for the last step.

CRF Model Integration Integration Method Heuristic method: (1) Add more orders, 1, 2, 3, to Mallet (2) Find the intersection with the lowest scores. (3) If no tagging result appears in the top 10 lists of all models, the best tagging result of Order 1 backward model will be selected simply (4) Find the intersection of two CRF++ models (5) Union CRF++ results and Mallet results

CRF Model Integration Integration Method

Conclusions (1) Different types of feature construction affect whether a CRF model is symmetric. (2) Backward parsing models enjoy a slight advantage over forward parsing according to the information gain analysis. (3) The combination of different models can achieve higher F-scores.

Reference This presentation is from: Hsu, C. N., Chang, Y. M., Kuo, C. J., Lin, Y. S., Huang, H. S., & Chung, I. F. (2008). Integrating high dimensional bi-directional parsing models for gene mention tagging. Bioinformatics, 24(13), i286-i294.

Thank you!