1. Semi-Supervised Learning over Text Tom M. Mitchell Machine Learning Department Carnegie Mellon University September 2006 Modified by Charles Ling

2. Statistical learning methods require LOTS of training data Can we use all that unlabelled text?

3. Outline Maximizing likelihood in probabilistic models –EM for text classification Co-Training and redundantly predictive features –Document classification –Named entity recognition –Theoretical analysis Sample of additional tasks –Word sense disambiguation –Learning HTML-based extractors –Large-scale bootstrapping: extracting from the web

4. Many text learning tasks Document classification. –f: Doc  Class –Spam filtering, relevance rating, web page classification,... –and unsupervised document clustering Information extraction. –f: Sentence  Fact, f: Doc  Facts Parsing –f: Sentence  ParseTree –Related: part-of-speech tagging, co-reference res., prep phrase attachment Translation –f: EnglishDoc  FrenchDoc

5. 1.Semi-supervised Document classification (probabilistic model and EM)

6. Document Classification: Bag of Words Approach aardvark0 about2 all2 Africa1 apple0 anxious0... gas1... oil1 … Zaire0

7. Supervised: Naïve Bayes Learner Train : For each class c j of documents 1. Estimate P(c j ) 2. For each word w i estimate P(w i | c j ) Classify (doc): Assign doc to most probable class * assuming words are conditionally independent, given class *

8. For code and data, see www.cs.cmu.edu/~tom/mlbook.html click on “Software and Data” www.cs.cmu.edu/~tom/mlbook.html Accuracy vs. # training examples

9. What if we have labels for only some documents? Y X1X1 X4X4 X3X3 X2X2 YX1X2X3X4 10011 00100 00010 ?0110 ?0101 Learn P(Y|X) EM: Repeat until convergence 1.Use probabilistic labels to train classifier h 2.Apply h to assign probabilistic labels to unlabeled data

10. From [Nigam et al., 2000]

11. E Step: M Step: w t is t-th word in vocabulary

12. Using one labeled example per class Words sorted by P(w|course) / P(w| : course)

13. 20 Newsgroups

15. Elaboration 1: Downweight the influence of unlabeled examples by factor New M step: Chosen by cross validation

16. Why/When will this work? What’s best case? Worst case? How can we test which we have?

17. EM for Semi-Supervised Doc Classification If all data is labeled, corresponds to supervised training of Naïve Bayes classifier If all data unlabeled, corresponds to mixture-of- multinomial clustering If both labeled and unlabeled data, it helps if and only if the mixture-of-multinomial modeling assumption is correct Of course we could extend this to Bayes net models other than Naïve Bayes (e.g., TAN tree) Other extensions: model negative class as mixture of N multinomials

18. 2. Using Redundantly Predictive Features (Co-Training)

19. Redundantly Predictive Features Professor Faloutsos my advisor

20. Co-Training Answer 1 Classifier 1 Answer 2 Classifier 2 Key idea: Classifier 1 and Classifier J must: 1. Correctly classify labeled examples 2. Agree on classification of unlabeled

21. CoTraining Algorithm #1 [Blum&Mitchell, 1998] Given: labeled data L, unlabeled data U Loop: Train g1 (hyperlink classifier) using L Train g2 (page classifier) using L Allow g1 to label p positive, n negative examps from U Allow g2 to label p positive, n negative examps from U Add these self-labeled examples to L

22. CoTraining: Experimental Results begin with 12 labeled web pages (academic course) provide 1,000 additional unlabeled web pages average error: learning from labeled data 11.1%; average error: cotraining 5.0% Typical run:

23. Co-Training for Named Entity Extraction (i.e.,classifying which strings refer to people, places, dates, etc.) Answer1 Classifier 1 Answer2 Classifier 2 I flew to New York today. New YorkI flew to ____ today [Riloff&Jones 98; Collins et al., 98; Jones 05]

24. One result [Blum&Mitchell 1998]: If –X1 and X2 are conditionally independent given Y –f is PAC learnable from noisy labeled data Then –f is PAC learnable from weak initial classifier plus unlabeled data CoTraining setting : wish to learn f: X  Y, given L and U drawn from P(X) features describing X can be partitioned (X = X1 x X2) such that f can be computed from either X1 or X2

25. Co-Training Rote Learner My advisor + - - pages hyperlinks - - - - + + - - - + + + + - - + + -

26. Co Training What’s the best-case graph? (most benefit from unlabeled data) What the worst case? What does conditional-independence imply about graph? x1x2 + - +

27. Expected Rote CoTraining error given m examples Where g is the jth connected component of graph of L+U, m is number of labeled examples j

28. How many unlabeled examples suffice? Want to assure that connected components in the underlying distribution, G D, are connected components in the observed sample, G S GDGD GSGS O(log(N)/  ) examples assure that with high probability, G S has same connected components as G D [Karger, 94] N is size of G D,  is min cut over all connected components of G D

29. PAC Generalization Bounds on CoTraining [Dasgupta et al., NIPS 2001] This theorem assumes X1 and X2 are conditionally independent given Y

30. Co-Training Theory Final Accuracy # unlabeled examples dependencies among input features # Redundantly predictive inputs # labeled examples Correctness of confidence assessments How can we tune learning environment to enhance effectiveness of Co-Training?  best: inputs conditionally indep given class, increased number of redundant inputs, …

31. Idea: Want classifiers that produce a maximally consistent labeling of the data If learning is an optimization problem, what function should we optimize? What if CoTraining Assumption Not Perfectly Satisfied? - + + +

32. Example 2: Learning to extract named entities I arrived in Beijing on Saturday. location? If: “I arrived in on Saturday.” Then: Location(X)

33. Co-Training for Named Entity Extraction (i.e.,classifying which strings refer to people, places, dates, etc.) Answer1 Classifier 1 Answer2 Classifier 2 I arrived in Beijing saturday. BeijingI arrived in __ saturday [Riloff&Jones 98; Collins et al., 98; Jones 05]

34. Bootstrap learning to extract named entities [Riloff and Jones, 1999], [Collins and Singer, 1999],... Iterations Initialization Australia Canada China England France Germany Japan Mexico Switzerland United_states locations in ?x South Africa United Kingdom Warrenton Far_East Oregon Lexington Europe U.S._A. Eastern Canada Blair Southwestern_states Texas States Singapore … operations in ?x Thailand Maine production_control northern_Los New_Zealand eastern_Europe Americas Michigan New_Hampshire Hungary south_america district Latin_America Florida... republic of ?x …...

35. Co-EM [Nigam & Ghani, 2000; Jones 2005] Idea: Like co-training, use one set of features to label the other Like EM, iterate –Assigning probabilistic values to unobserved class labels –Updating model parameters (= labels of other feature set)

36. CoEM applied to Named Entity Recognition [Rosie Jones, 2005], [Ghani & Nigam, 2000] Update rules:

37. CoEM applied to Named Entity Recognition [Rosie Jones, 2005], [Ghani & Nigam, 2000] Update rules:

38. CoEM applied to Named Entity Recognition [Rosie Jones, 2005], [Ghani & Nigam, 2000] Update rules:

40. [Jones, 2005] Can use this for active learning...

41. [Jones, 2005]

42. Idea: Want classifiers that produce a maximally consistent labeling of the data If learning is an optimization problem, what function should we optimize? What if CoTraining Assumption Not Perfectly Satisfied? - + + +

43. What Objective Function? Error on labeled examples Disagreement over unlabeled Misfit to estimated class priors

44. What Function Approximators? Same functional form as logistic regression Use gradient descent to simultaneously learn g1 and g2, directly minimizing E = E1 + E2 + E3 + E4 No word independence assumption, use both labeled and unlabeled data

45. Gradient CoTraining Classifying Capitalized sequences as Person Names 25 labeled 5000 unlabeled 2300 labeled 5000 unlabeled Using labeled data only Cotraining Cotraining without fitting class priors (E4).27.13.24 * sensitive to weights of error terms E3 and E4.11 *.15 * * Error Rates Eg., “Company president Mary Smith said today…” x1x2x1

46. Example 3: Word sense disambiguation [Yarowsky] “bank” = river bank, or financial bank?? Assumes a single word sense per document –X1: the document containing the word –X2: the immediate context of the word (‘swim near the __’) Successfully learns “context  word sense” rules when word occurs multiples times in documents.

47. Example 4: Bootstrap learning for IE from HTML structure [Muslea, et al. 2001] X 1 : HTML preceding the target X 2 : HTML following the target

48. Example Bootstrap learning algorithms: Classifying web pages [Blum&Mitchell 98; Slattery 99] Classifying email [Kiritchenko&Matwin 01; Chan et al. 04] Named entity extraction [Collins&Singer 99; Jones&Riloff 99] Wrapper induction [Muslea et al., 01; Mohapatra et al. 04] Word sense disambiguation [Yarowsky 96] Discovering new word senses [Pantel&Lin 02] Synonym discovery [Lin et al., 03] Relation extraction [Brin et al.; Yangarber et al. 00] Statistical parsing [Sarkar 01]

49. What to Know Several approaches to semi-supervised learning –EM with probabilistic model –Co-Training –Graph similarity methods –... –See reading list below Redundancy is important Much more to be done: –Better theoretical models of when/how unlabeled data can help –Bootstrap learning from the web (e.g. Etzioni, 2005, 2006) –Active learning (use limited labeling time of human wisely) –Never ending bootstrap learning? –...

50. Further Reading Semi-Supervised Learning, Olivier Chapelle, Bernhard Sch¨olkopf, and Alexander Zien (eds.), MIT Press, 2006. Semi-Supervised Learning Literature Survey, Xiaojin Zhu, 2006.Semi-Supervised Learning Literature Survey Unsupervised word sense disambiguation rivaling supervised methods D. Yarowsky (1995)Unsupervised word sense disambiguation rivaling supervised methods "Semi-Supervised Text Classification Using EM," K. Nigam, A. McCallum, and T. Mitchell, in Semi-Supervised Learning, Olivier Chapelle, Bernhard Sch¨olkopf, and Alexander Zien (eds.), MIT Press, 2006.Semi-Supervised Text Classification Using EM " Text Classification from Labeled and Unlabeled Documents using EM," K. Nigam, Andrew McCallum, Sebastian Thrun and Tom Mitchell. Machine Learning, Kluwer Academic Press, 1999. Text Classification from Labeled and Unlabeled Documents using EM " Combining Labeled and Unlabeled Data with Co-Training," A. Blum and T. Mitchell, Proceedings of the 1998 Conference on Computational Learning Theory, July 1998. Combining Labeled and Unlabeled Data with Co-Training Discovering Word Senses from Text Pantel & Lin (2002)Discovering Word Senses from Text Creating Subjective and Objective Sentence Classifiers from Unannotated Texts by Janyce Wiebe and Ellen Riloff (2005)Creating Subjective and Objective Sentence Classifiers from Unannotated Texts Graph Based Semi-Supervised Approach for Information Extraction by Hany Hassan, Ahmed Hassan and Sara Noeman (2006)Graph Based Semi-Supervised Approach for Information Extraction The use of unlabeled data to improve supervised learning for text summarization by MR Amini, P Gallinari (2002)The use of unlabeled data to improve supervised learning for text summarization

51. Further Reading Yusuke Shinyama and Satoshi Sekine. Preemptive Information Extraction using Unrestricted Relation DiscoveryPreemptive Information Extraction using Unrestricted Relation Discovery Alexandre Klementiev and Dan Roth. Named Entity Transliteration and Discovery from Multilingual Comparable Corpora.Named Entity Transliteration and Discovery from Multilingual Comparable Corpora Rion L. Snow, Daniel Jurafsky, Andrew Y. Ng. Learning syntactic patterns for automatic hypernym discoveryLearning syntactic patterns for automatic hypernym discovery Sarkar. (1999). Applying Co-training methods to Statistical Parsing. S. Brin, 1998. Extracting patterns and relations from the World Wide Web, EDBT'98Extracting patterns and relations from the World Wide Web O. Etzioni et al., 2005. "Unsupervised Named-Entity Extraction from the Web: An Experimental Study," AI Journal, 2005.Unsupervised Named-Entity Extraction from the Web: An Experimental Study