1. Automatic Eurovoc Indexing: Results and Evaluations Bruno Pouliquen Lang Tech group, JRC, European Commission Ispra-Italy http://www.jrc.cec.eu.int/langtech Addressing the Language Barrier Problem in the Enlarged EU Automating Eurovoc Descriptor Assignment

2. Contents Viewing the results –Browser –Exports –Validation interface Evaluation method –Test set –Precision/Recall –Evaluation interface Results

3. Browser For a given text: The original text The pre-processed text Keywords in the text (associates) Eurovoc descriptors manually assigned Eurovoc descriptors assigned automatically –With context Access to parallel texts

4. Browser example: COMMISSION DECISION of 8 September 1997 on the temporary suspension of imports of pistachios and certain products derived from pistachios originating in or consigned from Iran (Text with EEA relevance) (97/613/EC) THE COMMISSION OF THE EUROPEAN COMMUNITIES, Having regard to the Treaty establishing the European Community, Having regard to Council Directive 93/43/EEC of 14 June 1993 on the hygiene of foodstuffs (1), and in particular Article 10 thereof, Whereas pistachios originating in or consigned from Iran are in many cases contaminated with excessive levels of Aflatoxin B1; Whereas the Scientific Committee for Food has noted that Aflatoxin B1, even at extremely low doses, causes cancer of the liver and in addition it is genotoxic; Whereas this constitutes a serious threat to public health within the Community and it is imperative to adopt urgently protective measures at Community level; Whereas, in the absence, at this time, of sanitary guarantees from the Iranian authorities, it is necessary to suspend imports of pistachios and certain products derived from pistachios originating in or consigned from Iran; …

5. Pre-processed text

6. Keywords (associates) in text Histogram Keywords occurring in the text

7. Eurovoc descriptors assigned Descriptors assigned manually Descriptors assigned automatically

8. Browser: online demo Resolution on human rights in Ethiopia Commission Decision (import of meat products…)

9. Export of the results XML file containing the assignment PRESIDENCY OF THE EC COUNCIL EUROPEAN UNION PRESIDENT SOCIAL POLICY PRINCIPLE OF SUBSIDIARITY...

10. Validation interface: overview “Financial Instrument for Fisheries Guidance” is in the text This text was previously indexed with this descriptor “fish”, “fisherman”, “fishery”, “conservation” and “fishery_resources” are in the text

11. Validation interface: example of good assignment

12. Validation interface: example of bad assignment on a small text Strangely, “Austria” was manually assigned “UN convention” was manually assigned

13. Validation interface: other example of bad assignment...

14. Evaluation method A test set is built –Not used for training –Should be representative After training, we compare automatically the manually assigned descriptors to the automatically assigned ones –Depending on the rank (number of descriptors) –Depending on the various parameters and formulae Use precision/recall

15. Evaluation results Use precision/recall (here: English) RankPrecisionRecallPrec RTRec RT F1- measure 176.41814.56082.89015.88624.45 268.61725.80076.46328.05137.49 361.82034.53171.01137.82344.31 457.11442.27967.50945.85048.58 551.48947.49663.20951.27949.41 647.01551.84359.42755.77649.31 743.36455.68756.13059.45348.75 840.02758.66053.14762.37847.58 936.87960.78250.32564.73845.90 F1-Measure: combines precision and recall (Harmonic average)

16. Evaluation interface Graph showing precision/recall/F-measure depending on the number of descriptors

17. Results across languages With pre-processing (French=> only stop words)Without pre-processing

18. Validation: expert judgment Expert judgment on automatic assignment: – ‘ G ’ for good descriptor –‘NT’ for good, but it would have been better to use a NT instead –‘BT’ for good, but its BT would have been better –‘?’ for un­known /not possible to make a judgement in the time available –‘B’ for clearly bad –‘S’ for semantically related, but wrong.

19. Manual Evaluation of the Assignment

20. Manual Evaluation Manual Evaluation of manual assignment Manual Evaluation - Overview

21. Manual Evaluation of Automatic Assignment Correct descriptors compared to benchmark of manual assignment English: 65 / 78 = 83% Spanish: 69 / 87 = 80%

22. Online demo Validation: list of texts to be validated Some examples: –In FrenchIn French –Already validated in SpanishAlready validated in Spanish –With the manual assignmentWith the manual assignment