Automatic Language Identification – A Syntactic Approach Mahesh Soundalgekar November 23, 2018 CFILT, IIT Bombay

The Road Map Introduction System Architecture Classification Approaches Experimental Results Summary and Future Work November 23, 2018 CFILT, IIT Bombay

Introduction Goal : Efficiently crawl Web pages in a given language; Marathi in our case Different languages use the same Devanagari script E.g Marathi, Sanskrit and Hindi Necessity to accurately distinguish one language from others We take a syntactic approach to solve this problem, which has given us excellent results on training data of 2MB with test data of 10 MB November 23, 2018 CFILT, IIT Bombay

Appropriate Encoding Converter System Architecture HTML Documents in different encodings such as Xdvng, DV-TTYogesh HTML to ASCII Plain Text + Font Information Appropriate Encoding Converter Plain Text in ISCII Encoding Classifier Classification Results November 23, 2018 CFILT, IIT Bombay

Classification Approaches Most Frequently Occurring Common Words e.g. English : the, an, is, at,a etc N-Grams (Most Frequent Character Sequences) Bi-grams: th, ’s, re, en Tri-grams: the, ing, ion, Quad-grams: tion as in classification, association, gratification etc. November 23, 2018 CFILT, IIT Bombay

Important Factors Size of the Training Data – Important to capture the syntactic essence of a language Domains of Training Data – Usages vary from domain to domain, author to author Size of the Test Data – Small test data may not contain enough information for classification Requirement of linguistic knowledge for common words approach November 23, 2018 CFILT, IIT Bombay

Classifier Architecture Training Samples Test Document Generate Profile Generate Profiles Category Profiles Document Profile Measure Profile Distances Find minimum Distance Identify category November 23, 2018 CFILT, IIT Bombay

Common Words Approach List of selected common words Matched with the test documents Closest match will give the language of the document Advantages: Intuitive Computationally Efficient Space Efficient November 23, 2018 CFILT, IIT Bombay

Top 5 Marathi Common Words ´É +ÉÎhÉ +É½ä ªÉÉ iÉä November 23, 2018 CFILT, IIT Bombay

N-Grams Approach JAVA Bi-grams: _J, JA, AV, VA, A_ Tri-grams: _JA, JAV, AVA, VA_, A__ Quad-grams: _JAV, JAVA, AVA_, VA__, A___ ¨ÉniÉ Bi-grams: _¨É, ¨Én, , niÉ, iÉ_ Tri-grams: _¨Én, ¨ÉniÉ, niÉ_, iÉ__ November 23, 2018 CFILT, IIT Bombay

Measuring Distances Out_of_Place () A ER ING AND ON AR AND ER ED ON max_value 2 1 Max_value Category profile sorted in descending order Test profile sorted in descending order Distance =3 + 2* max_value November 23, 2018 CFILT, IIT Bombay

Extensions to N-Grams Method Lowest Granularity +ÉÊniªÉ = + + É + Ê + n + iÉ + ªÉ Letter Granularity +ÉÊniªÉ = +É + Ên + iÉ + ªÉ Conjunct Granularity +ÉÊniªÉ = +É + Ên + iªÉ November 23, 2018 CFILT, IIT Bombay

Experimental Training Setup Language Total size of pages in KB No. of Pages Average size of a page in KB Marathi 700 46 15.2 Hindi 600 24 25 Sanskrit 560 19 29.5 November 23, 2018 CFILT, IIT Bombay

Category Profiles Generated through Training Language No. of handpicked Common Words No. of N-Grams in Atomic Approach Letter Approach Conjunct Approach Marathi 25 37633 63596 63580 Hindi 15450 26886 26865 Sanskrit 21 24119 45380 49368 November 23, 2018 CFILT, IIT Bombay

Classification Results Language Common Words Atomic Approach Letter Approach Conjunct Approach Marathi 91% 95% 100% Hindi 93% 80% 92% Sanskrit 86% 50% November 23, 2018 CFILT, IIT Bombay

Summary and Future Work Good results have been obtained through syntactic classification Common words technique is computationally most efficient, but with a lesser accuracy Our extensions to N-Grams give the desired accuracy N-grams technique is robust to syntax errors N-Grams technique does not require linguistic knowledge We will be Using language identification techniques to identify a good starting set of pages for crawling activities for the general purpose search engine November 23, 2018 CFILT, IIT Bombay