1. Omphalos Session

2. Omphalos Session Programme Design & Results 25 mins Award Ceremony 5 mins Presentation by Alexander Clark25 mins Presentation by Georgios Petasis 10 mins Open Discussion on Omphalos and GI competition20 mins

3. Omphalos : Design and Results Brad Starkie François Coste Menno van Zaanen

4. Contents Design of the Competition –A complexity measure for GI Results Conclusions

5. Aims Promote new and better GI algorithms A forum to compare GI algorithms Provide an indicative measure of current state-of-the-art.

6. Design Issues Format of training data Method of evaluation Complexity of tasks

7. Training Data Plain Text or Structured Data –Bracketed, Partially bracketed, Labelled, Unlabelled (+ve and –ve data) or (+ve data only) Plain Text,(+ve and –ve) and (+ve only) –Similar to Abbadingo –Placed fewest restrictions on competitors

8. Method of Evaluation Classification of unseen examples Precision and Recall Comparison of derivation trees Classification of unseen examples –Similar to Abbadingo –Placed fewest restrictions on competitors

9. Complexity of the Competition Tasks Learning task should be sufficiently difficult. –Outside the current state-of-the-art, but not too difficult Ideally provable that the training sentences are sufficient to identify the target language

10. Three axes of difficulty Complexity of the underlying grammar +ve/-ve or +ve only. Similarity between -ve and +ve examples.

11. Complexity Measure of GI Created a model of the GI based upon a brute force search (Non polynomial) Complexity measure = size of the hypothesis space created when presented with a characteristic set.

12. Hypothesis Space for GI All CFGs can be converted to Chomsky Normal Form. For any sentence there are a finite number of unlabelled derivations given CNF –Finite number of labelled derivation trees The grammar can be reconstructed given sufficient number of derivation trees All possible labelled derivation trees corresponds to all possible CNF grammars given the maximum number of nterms Solution: calculate max number of nterms and create all possible grammars

13. BruteForceLearner Given the positive examples construct all possible grammars Discard any grammars that generate any negative sentences Randomly select a grammar from hypothesis set

14. Characteristic Set of Positive Sentences Put the grammar into minimal CNF form –If a single rule is removed one or more sentences can't be derived For each rule add a sentence that can only be derived using that rule –Such a sentence exists if G in minimal form When presented with this set, one of the hypothesis grammars is correct

15. Characteristic set of Negative Sentences. Given G calculate positive sentences Construct hypothesis set For each hypothesis H  G, L(H)  L(G) add + sentence s | s  L(G) but s  L(H) For each hypothesis H  G, L(H)  L(G) add - sentence s | s  L(H) but s  L(G) Generating -ve data according to this technique requires exponential time – Therefore cannot be used to generate –ve data in Omphalos.

16. Creation of the Target Grammars Benchmark probs identified in literature –Stolcke-93,Nakamura-02,Cook-76,Hopcroft-01 Number of nterms, terms and rules were selected Randomly generated grammars, useless rules removed, CF constructs (center recursion) added A characteristic set of sentences was generated, and complexity measured To test if deterministic, LR(1) tables created using Bison For non-deterministic grammar non-deterministic constructs added

17. Creation of positive data Characteristic set generated from grammar Additional training examples added –Size of training set 10  20  size of characteristic set Longest training example was shorter than the longest test

18. Creation of negative data Not guaranteed to be sufficient Originally randomly created (bad idea) For probs 6a  10 regular equivalents to grammars constructed and negative data could be generated from regular equivalent to CFG –Nederhof-00 Center recursion expanded to a finite depth Vs true center recursion Equal number of positive and negative examples in the test sets

19. Participation Omphalos 1st page: ~ 1000 hits from 270 domains –Attempted to discard crawlers and bots hits –All continents except 2 Data sets : downloaded by 70 different domains Oracle: 139 label submissions by 8 contestants (4) –Short test sets: 76 submissions –Large test sets: 63 submissions

20. Results

21. Techniques Used. Prob 1 –Solved by hand. Probs 3, 4, 5, and 6 –Pattern matching using n-grams. –Generated its own negative data –the majority of randomly generated strings would not be contained within the language. Probs 2, 6.2, 6.4 –Distributional Clustering and ABL

22. Conclusions The way in which negative data is created is crucial to judging performance of competitors entries

23. Review of Aims Promote development of new and better GI algorithms –Partially achieved A forum to compare different GI algorithms –Achieved Provide an indicative measure of the state- of-the-art. –Achieved