SANSKRIT ANALYZING SYSTEM Manji Bhadra, Surjit Kumar Singh, Sachin Kumar, Subash, Diwakar Mishra Muktanand Agrawal, R.Chandrashekar, Sudhir K Mishra, Girish Nath Jha 3rd ISCLS, Hyderabad

Introduction It is an attempt towards analysis of laukika Sanskrit Major goal is to build a machine translation system from Sanskrit to other Indian language. The modules have been developed separately We need to integrate these modules We need to evaluate these modules 3rd ISCLS, Hyderabad

Introduction The system accepts full text inputs in Devanagari Unicode (UTF-8). It supports two IMEs - Baraha and J-IME. It has two major components- the shallow parser the kraka analyzer 3rd ISCLS, Hyderabad

Shallow parser The modules are as follows- sandhi analyzer samsa analyzer * subanta analyzer gender analyzer kdanta analyzer taddhita analyzer* tianta analyzer POS tagger * Modules are under development 3rd ISCLS, Hyderabad

How does it work Show example 3rd ISCLS, Hyderabad

Our platform Java servlet based web application and services. Java, JSP for frontend. Unicode input/output with flatfiles, RDBMS (MS-SQL server 2005) MS-JDBC driver for connectivity Apache-Tomcat for web server Javascript IME for unicode output with Itrans input 3rd ISCLS, Hyderabad

Sandhi analyzer Sandhi processing is critical for any further processing of Sanskrit. Without sandhi-vichheda it is not possible to get the word constituents for analysis. At present, our sandhi analyzer does only vowel sandhi splitting. The consonant splitting is under development. Our goal is to be able to parse a very complex string with potentially all kinds of sandhi 3rd ISCLS, Hyderabad

Sandhi analyzer input Sanskrit text ↓ viccheda eligibility tests (pre-processing) subanta processing search of sandhi marker and sandhi patterns (‌‌sandhi rule base) generate possible solutions (result generator) search the lexicon (to parse the vibhakti of first segment, if any) output (segmented text) 3rd ISCLS, Hyderabad

Sandhi analyzer 1. tokenize by space (words) 2. preprocess (exclude puncts) -> puncts marked 3. check example base - if found stop 4. check subanta (it checks avyayas, verbs as well) -> pratipadikas -> avyayas marked -> verbs marked 3rd ISCLS, Hyderabad

Sandhi Analyzer 5. check pratipadika list -> if found then dont process for Sandhi -> if not found then start sandhi processing 6. search of sandhi marker and sandhi patterns 7. generate possible solutions 8. search the lexicon 9. subanta processing (to parse the vibhakti of first segment, if any) 10. output (segmented text) 3rd ISCLS, Hyderabad

Live demo from JNU server Demo from localhost 3rd ISCLS, Hyderabad

Subanta analyzer Isolating the inflections and obtaining nominal bases and its case terminations is essential for morph analysis. The system has Unicode Devanagari input/output mechanism and accepts complete text as well 3rd ISCLS, Hyderabad

Subanta analyzer INPUT TEXT ↓ PRE-PROCESSOR VERB DATABASE  LIGHT POS TAGGING  AVYAYA DATABASE SUBANTA RECOGNIZERVIBHAKTI DATABASE SUBANTA RULES SUBANTA ANALYZER SANDHI RULES SUBANTA ANALYSIS 3rd ISCLS, Hyderabad

Subanta Analyzer Works on a subanta rulebase and example-base Subanta eligibility check fixed lists (punctuations, avyayas, verbs) If found  tag Else mark them SUBANTA Check it in dictionary If found store separately Else start subanta processing 3rd ISCLS, Hyderabad

Subanta Analyzer check example-base Template search If found  tag else continue Template search Evaluate string as per set templates Split it in parts and match the viccheda patterns If found  obtain corresponding analysis Else tag the input SUBANTA 3rd ISCLS, Hyderabad

Live demo from JNU server demo from localhost 3rd ISCLS, Hyderabad

Kdanta Analyzer 3rd ISCLS, Hyderabad

Kdanta Analysis The process of kdanta analysis mechanism is divided into two sections - recognition and analysis. The kdanta recognition starts by an exclusion process. The verb forms, avyayas and punctuations are excluded by running POS tagger The nominal bases are obtained by the subanta analyzer These nominal bases are then checked in fixed lists. This may result in some of the subantas being marked for kdanta. The remaining subantas are sent to the kdanta recognizer and analyzer system for recognition and analysis using following steps –    3rd ISCLS, Hyderabad

Kdanta Analysis check the kdanta database, annotated corpus and kdanta-tagged Monier Williams Sanskrit Digital Dictionary (MWSDD). the subantas still untagged for kdanta are sent to the rule base for kdanta checking. there may still remain an untagged kdanta subanta. This will count as failure of the system. अधिकारी["अधिकारिन्","अधि+डुकृञ्","इनि","noun_m"]_KR 3rd ISCLS, Hyderabad

Tianta Analysis The methodology is a mix of using verb database and reverse Paninian processing pre-processing take token by token confirm the verb (dict, check suffixes), ignore others Check database If not found  start analysis analyze suffixes evaluate remaining string for base (dict check for bases) result 3rd ISCLS, Hyderabad

POS Tagger Rule-based tagger is developed for Sanskrit Language. There are three kinds of tags in this tagset- Word class main tags, feature sub-tags, punctuation tags. The tag as a whole is a combination of word class main tag with feature sub-tags separated by an underscore All the tags bear Sanskrit names with letter-digit acronymic in Roman script Tagset (JNU server) Tagset (localhost) 3rd ISCLS, Hyderabad

POS Tagger Input text Pre processing Fixed list tagger Morph analyzer Disambiguator* Result normalizer Display tagged text 3rd ISCLS, Hyderabad

Gender Analyzer Along with the information of vibhakti and number, it is also necessary to have information of gender. In Sanskrit there is agreement within noun phrase in terms of vibhakti, number and gender. While translating a Sanskrit sentence into Hindi, it is necessary to know what would be collocational gender of the sentence, otherwise the whole translation may be wrong. 3rd ISCLS, Hyderabad

Gender Analyzer Input Sanskrit text Un-anlayzed Text Lexical lookup Un-anlayzed Text Application of Subanta Analyzer Lexical lookup Subanta Analyzed Text Un-analyzed text Application of rulebase Check gender agreement within a noun phrase Suggest the gender of the noun phrase for Hindi translation 3rd ISCLS, Hyderabad

live demo from JNU server demo from localhost 3rd ISCLS, Hyderabad

Kraka Analyzer VERB ID VERB ANALYSIS NON—VERB ID SUBANTA ANALYSIS   VERB ID VERB ANALYSIS NON—VERB ID SUBANTA ANALYSIS KK CHECK* KRAKA RULES* SPECIAL CONDITIONS KRAKA ASSIGNMENT 3rd ISCLS, Hyderabad

Conclusion The authors in this paper have presented an ongoing work for developing a complete SAS. Currently, the SAS has some modules partially developed and some under development. Significant future additions will be the Taddhita, samasa modules ambiguity resolution modules System integration module Evaluation module 3rd ISCLS, Hyderabad

Thank You! http://sanskrit.jnu.ac.in 3rd ISCLS, Hyderabad