1. 1 Information Extraction using HMMs Sunita Sarawagi

2. 2 IE by text segmentation Source: concatenation of structured elements with limited reordering and some missing fields Example: Addresses, bib records House number BuildingRoadCity Zip 4089 Whispering Pines Nobel Drive San Diego CA 92122 P.P.Wangikar, T.P. Graycar, D.A. Estell, D.S. Clark, J.S. Dordick (1993) Protein and Solvent Engineering of Subtilising BPN' in Nearly Anhydrous Organic Media J.Amer. Chem. Soc. 115, 12231-12237. Author Year TitleJournal Volume Page State

3. 3 Hidden Markov Models Doubly stochastic models Efficient dynamic programming algorithms exist for Finding Pr(S) The highest probability path P that maximizes Pr(S,P) (Viterbi) Training the model (Baum-Welch algorithm) S2S2 S4S4 S1S1 0.9 0.5 0.8 0.2 0.1 S3S3 ACAC 0.6 0.4 ACAC 0.3 0.7 ACAC 0.5 ACAC 0.9 0.1

4. 4 Input features  Content of the element  Specific keywords like street, zip, vol, pp,  Properties of words like capitalization, parts of speech, number?  Inter-element sequencing  Intra-element sequencing  Element length  External database  Dictionary words  Semantic relationship between words  Frequency constraints

5. 5 IE with Hidden Markov Models Probabilistic models for IE Title Journal Author 0.9 0.5 0.8 0.2 0.1 Transition probabilities Year ABCABC 0.6 0.3 0.1 XBZXBZ 0.4 0.2 0.4 YACYAC 0.1 0.8 Emission probabilities dddd dd 0.8 0.2

6. 6 HMM Structure Naïve Model: One state per element Nested model Each element another HMM

7. 7 Comparing nested models Naïve: Single state per tag Element length distribution: a, a 2, a 3,… Intra-tag sequencing not captured Chain: Element length distribution: Each length gets its own parameter Intra-tag sequencing captured Arbitrary mixing of dictionary, Eg. “California York” Pr(W|L) not modeled well. Parallel path: Element length distribution: each length gets a parameter Separates vocabulary of different length elements, (limited bigram model)

8. 8 Embedding a HMM in a state

9. 9 Bigram model of Bikel et al. Each inner model a detailed bigram model First word: conditioned on state and previous state Subsequent words conditioned on previous word and state Special “start” and “end” symbols that can be thought Large number of parameters (Training data order~60,000 words in the smallest experiment) Backing off mechanism to previous simpler “parent” models (lambda parameters to control mixing)

10. 10 Separate HMM per tag Special prefix and suffix states to capture the start and end of a tag S2S2 S4S4 S1S1 Prefix Suffix Road name S2S2 S4S4 S1S1 S3S3 Prefix Suffix Building name

11. 11 HMM Dictionary For each word (=feature), associate the probability of emitting that word Multinomial model Features of a word, example, part of speech, capitalized or not type: number, letter, word etc Maximum entropy models (McCallum 2000), other exponential models Bikel: pairs

12. 12 Feature Hierarchy

13. 13 Learning model parameters When training data defines unique path through HMM Transition probabilities Probability of transitioning from state i to state j = number of transitions from i to j total transitions from state i Emission probabilities Probability of emitting symbol k from state i = number of times k generated from i number of transition from I When training data defines multiple path: A more general EM like algorithm (Baum-Welch)

14. 14 Smoothing Two kinds of missing symbols: Case-1: Unknown over the entire dictionary Case-2: Zero count in some state Approaches: Laplace smoothing: ki + 1 m + |T| Absolute discounting P(unknown) proportional to number of distinct tokens P(unknown) = (k’) x (number of distinct symbols) P(known) = (actual probability)-(k’), k’ is a small fixed constant, case 2 smaller than case 1

15. 15 Smoothing (Cont.) Smoothing parameters derived from data Partition training data into two parts Train on part-1 Use part-2 to map all new tokens to UNK and treat it as new word in vocabulary OK for case-1, not good for case-2. Bikel et al use this method for case-1. For case-2 zero counts are backed off to 1/(Vocab-size)

16. 16 Using the HMM to segment Find highest probability path through the HMM. Viterbi: quadratic dynamic programming algorithm House otot Road City Pin 115 Grant street Mumbai 400070 House Road City Pin 115 Grant ……….. 400070 otot House Road City Pin House Road Pin

17. 17 Most Likely Path for a Given Sequence The probability that the path is taken and the sequence is generated: transition probabilities emission probabilities

18. 18 Example A 0.1 C 0.4 G 0.4 T 0.1 A 0.4 C 0.1 G 0.1 T 0.4 beginend 0.5 0.2 0.8 0.4 0.6 0.1 0.9 0.2 0.8 05 4 3 2 1 A 0.4 C 0.1 G 0.2 T 0.3 A 0.2 C 0.3 G 0.3 T 0.2

19. 19 Finding the most probable path: the Viterbi algorithm define to be the probability of the most probable path accounting for the first i characters of x and ending in state k we want to compute, the probability of the most probable path accounting for all of the sequence and ending in the end state can define recursively can use dynamic programming to find efficiently

20. 20 Finding the most probable path: the Viterbi algorithm initialization:

21. 21 The Viterbi algorithm recursion for emitting states (i =1…L): keep track of most probable path

22. 22 The Viterbi algorithm to recover the most probable path, follow pointers back starting at termination:

23. 23 Database Integration Augment dictionary Example: list of Cities Assigning probabilities is a problem Exploit functional dependencies Example Santa Barbara -> USA Piskinov -> Georgia

24. 24 2001 University Avenue, Kendall Sq. Piskinov, Georgia 2001 University Avenue, Kendall Sq., Piskinov, Georgia House number Road Name CityState Area 2001 University Avenue, Kendall Sq., Piskinov, Georgia House number Road Name AreaCountryCity

25. 25 Frequency constraints Including constraints of the form: the same tag cannot appear in two disconnected segments Eg: Title in a citation cannot appear twice Street name cannot appear twice Not relevant for named-entity tagging kinds of problems

26. 26 Constrained Viterbi Original Viterbi Modified Viterbi ….

27. 27 Comparative Evaluation Naïve model – One state per element in the HMM Independent HMM – One HMM per element; Rule Learning Method – Rapier Nested Model – Each state in the Naïve model replaced by a HMM

28. Results: Comparative Evaluation The Nested model does best in all three cases (from Borkar 2001) Datasetinsta nces Elem ents IITB student Addresses 238817 Company Addresses 7696 US Addresses 7406

29. 29 Results: Effect of Feature Hierarchy Feature Selection showed at least a 3% increase in accuracy

30. 30 Results: Effect of training data size HMMs are fast Learners. We reach very close to the maximum accuracy with just 50 to 100 addresses

31. 31 HMM approach: summary Inter-element sequencing Intra-element sequencing Element length Characteristic words Non-overlapping tags  Outer HMM transitions  Inner HMM  Multi-state Inner HMM  Dictionary  Global optimization