1. PASCAL P ASCAL C HALLENGE ON I NFORMATION E XTRACTION & M ACHINE L EARNING Neil Ireson Local Challenge Coordinator Web Intelligent Group Department of Computer Science University of Sheffield

2. PASCAL Organisers Sheffield – Fabio Ciravegna UCD Dublin – Nicholas Kushmerick ITC-IRST – Alberto Lavelli University of Illinois – Mary-Elaine Califf FairIsaac – Dayne Freitag Website http://tyne.shef.ac.uk/Pascal

3. PASCAL Outline Challenge Goals Data Tasks Participants Results on Each Task Conclusion

4. PASCAL Goal : Provide a testbed for comparative evaluation of ML-based IE Standardised data Partitioning Same set of features –Corpus preprocessed using Gate –No features allowed other than the ones provided Explicit Tasks Standard Evaluation Provided independently by a server For future use Available for further test with same or new systems Possible to publish and new corpora or tasks

5. PASCAL Data (Workshop CFP) 2005 1993 2000 Training Data 400 Workshop CFP Testing Data 200 Workshop CFP

6. PASCAL Data (Workshop CFP) 2005 1993 2000 Training Data 400 Workshop CFP Testing Data 200 Workshop CFP Set0 Set1 Set2 Set3

7. PASCAL Data (Workshop CFP) 2005 1993 2000 Training Data 400 Workshop CFP Testing Data 200 Workshop CFP Set0 Set1 Set2 Set3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9

8. PASCAL Data (Workshop CFP) 2005 1993 2000 Training Data 400 Workshop CFP Testing Data 200 Workshop CFP Enrich Data 1 250 Workshop CFP Enrich Data 2 250 Conference CFP WWW

9. PASCAL Preprocessing GATE –Tokenisation –Part-Of-Speech –Named-Entities Date, Location, Person, Number, Money

10. PASCAL Annotation Exercise 4+ months Initial consultation 40 documents – 2 annotators Second consultation 100 documents – 4 annotators Determine annotation disagreement Full annotation – 10 annotators Annotators Christopher Brewster Sam Chapman Fabio Ciravegna Claudio Giuliano Jose Iria Ashred Khan Vita Lanfranchi Alberto Lavelli Barry Norton

11. PASCAL

12. Annotation Slots Training CorpusTest corpus workshopname54311.8%24510.8% acronym56612.3%24310.7% homepage3678.0%2159.5% location45710.0%2249.9% date58612.8%32614.3% paper submission date59012.9%31613.9% notification of acceptance date3918.5%1908.4% camera-ready copy date3557.7%1637.2% conferencename2044.5%904.0% acronym4209.2%1878.2% homepage1042.3%753.3% Total4583100.0%2274100.0%

13. PASCAL Evaluation Tasks Task1 - ML for IE: Annotating implicit information –4-fold cross-validation on 400 training documents –Final Test on 200 unseen test documents Task2a - Learning Curve: –Effect of increasing amounts of training data on learning Task2b - Active learning: Learning to select documents –Given seed documents select the documents to add to training set Task3a - Enriched Data: –Same as Task1 but can use the 500 unannotated documents Task3b - Enriched & WWW Data: –Same as Task1 but can use all available unannotated documents

14. PASCAL Evaluation Precision/Recall/F 1 Measure MUC Scorer Automatic Evaluation Server Exact matching Extract every slot occurrence

15. PASCAL Participants Participant ML 4-fold X-validationTest Corpus 12a2b3a3b12a2b3a3b Amilcare (Sheffield, UK)LP 2 2211111 Bechet (Avignon, France)HMM2122 Canisius (Netherlands)SVM, IBL11 Finn (Dublin, Ireland)SVM11 Hachey (Edinburgh, UK)MaxEnt, HMM11 ITC-IRST (Italy)SVM331 Kerloch (France)HMM2232 Sigletos (Greece)LP 2, BWI, ?13 Stanford (USA)CRF11 TRex (Sheffield, UK)SVM2 Yaoyong (Sheffield, UK)SVM333333 Total1584002010511

16. PASCAL Task1 Information Extraction with all the available data

17. PASCAL Task1: Test Corpus

18. PASCAL Task1: Test Corpus

19. PASCAL Task1: 4-Fold Cross-validation

20. PASCAL Task1: 4-Fold & Test Corpus

21. PASCAL Task1: Slot FMeasure

22. PASCAL Best Slot FMeasures Task1: Test Corpus

23. PASCAL Slot Recall: All Participants

24. PASCAL Task 2a Learning Curve

25. PASCAL Task2a: Learning Curve FMeasure

26. PASCAL Task2a: Learning Curve Precision

27. PASCAL Task2a: Learning Curve Recall

28. PASCAL Task 2b Active Learning

29. PASCAL Active Learning (1) 400 Potential Training Documents 200 Test Documents

30. PASCAL Active Learning (1) 360 Potential Training Documents 40 Selected Training Document 200 Test Documents Select Test

31. PASCAL Active Learning (2) 360 Potential Training Documents 200 Test Documents Subset0 40 Training Documents Extract

32. PASCAL Active Learning (2) 320 Potential Training Documents 40 Selected Training Documents 200 Test Documents Select Test Subset0 40 Training Documents

33. PASCAL Active Learning (3) 320 Potential Training Documents 200 Test Documents Subset0,1 80 Training Documents Extract

34. PASCAL Active Learning (3) 280 Potential Training Documents 40 Selected Training Documents 200 Test Documents Select Test Subset0,1 80 Training Documents

35. PASCAL Task2b: Active Learning Amilcare –Maximum divergence from expected number of tags. Hachey –Maximum divergence between two classifiers built on different feature sets. Yaoyong (Gram-Schmidt) –Maximum divergence between example subset.

36. PASCAL Task2b: Active Learning Increased FMeasure over random selection

37. PASCAL Task 3 Semi-supervised learning (not significant participation)

38. PASCAL Conclusions (Task1) Top three (4) systems use different algorithms –Amilcare : Rule Induction –Yaoyong : SVM –Stanford : CRF –Hachey : HMM

39. PASCAL Conclusions (Task1: Test Corpus) Same algorithms (SVM) produced different results

40. PASCAL Conclusions (Task1: 4-fold Corpus) Same algorithms (SVM) produced different results

41. PASCAL Conclusions (Task1) Task 1 –Large variation on slot performance Good performance on: –“Important” dates and Workshop homepage –Acronyms (for Amilcare) Poor performance on: –Workshop name and location –Conference name and homepage

42. PASCAL Conclusion (Task2 & Task3) Task 2a: Learning Curve –Systems’ performance is largely as expected Task 2b: Active Learning –Two approaches, Amilcare and Hachey, showed benefits Task 3: Enrich Data –Not sufficient participation to evaluate use of enrich data

43. PASCAL Future Work Performance differences: –Systems: what determines good/bad performance –Slots: different systems were better/worse at identifying different slots Combine approaches Active Learning Enrich data –Overcoming the need for annotated data Extensions –Data: Use different data sets and other features, using (HTML) structured data –Tasks: Relation extraction

44. PASCAL Why is Amilcare Good?

45. PASCAL Contextual Rules

46. PASCAL Contextual Rules

47. PASCAL Rule Redundancy