Semi-automatic glossary creation from learning objects Eline Westerhout & Paola Monachesi

Overview The LT4eL project Detecting definitions –eLearning domain –Types of definitory contexts –Grammar approach –Machine learning approach Qualitative evaluation Conclusions Future work

LT4eL - Language Technology for eLearning Start date: 1 December 2005 Duration: 30 months Partners: 12 Languages: 8 WWW:

Tasks Creation of an archive of learning objects Semi-automatic metadata generation driven by NLP tools: –Keyword extractor Support the development of glossaries –Definition extractor Enhancing eLearning with semantic knowledge: ontologies Integration of functionalities in LMS Validation of new functionalities in LMS

Extraction of definitions within eLearning Definition extraction: –question answering –building dictionaries from text –ontology learning Challenges within our project: –corpus –size of LOs

Types is_def: ‘Gnuplot is a program for drawing graphs’ verb_def: ‘eLearning comprises resources and applications that are available via the internet and provide creative possibilities to improve the learning experience’ punct_def ‘Passes: plastic cards equipped with a magnetic strip, that [...] gets access to certain facilities. ’ pron_def ‘Dedicated readers. These are special devices, developed with the exclusive goal to make it possible to read e-books.’

Grammar approach General Example Results

Identification of definitory contexts Make use of the linguistic annotation of LOs (part-of- speech tags) Domain: computing Use of language specific grammars Workflow –Searching and marking definitory contexts in LOs (manually) –Drafting local grammars on the basis of these examples –Apply the grammars to new LOs

Een vette letter is een letter die zwarter wordt afgedrukt dan de andere letters.

Een vette letter is een letter die zwarter wordt afgedrukt dan de andere letters.

Results (grammar)

Machine learning General Features & Configurations Results

General information Naive Bayes classifier Weka Data set: –is-definitions: 77 / 274 –punct-definitions: 45 / fold cross validation

Features Text properties: bag-of-words, bigrams, and bigram preceding the definition Syntactic properties: type of determiner within the defined term (definite, indefinite, no determiner) Proper nouns: presence of a proper noun in the defined term cf. Fahmi & Bouma, 2006

Configurations

Results – is_def (final)

Results – punct_def (final)

Final results precision: + (50 % and 40 %) recall: - (20 % and 30 %) F-score: + (30 % and 25 %)

Qualitative evaluation Scenario based evaluation, 1 st cycle Tested tutors (3) and students (6) Results: –usefulness glossary: useful (tutors: 2/3, students: 7/7) –usefulness search method 'Definitions': useful (students: 7/7) –performance: different (tutors: 1 ok, 1 not ok, 1 neutral)

Conclusions Recall: ok with pattern based grammar Precision: can be improved with machine learning approach Trade-off between recall and precision –Only PB: good recall, bad precision –PB & ML: better precision, lower recall For our purpose: recall might be more important than precision Size of objects also important

Future work Machine learning: try different features use other classifiers extend to all types of definitions Qualitative evaluation 2 nd cycle scenario based evaluation => focus more on preference user

Results – is_def (ML)

Results – punct_def (ML)