1. A Virtual Character based Italian Sign Language Dictionary Instructor: Dr. Nadeem Ahmad Chauhdry Dual PhD (Politecnico di Torino, Italy & University of Potsdam, Germany) HOD, CS & IT Department The University of Lahore Defence Road Campus, Lahore. Email: nadeem.ahmad@cs.uol.edu.pknadeem.ahmad@cs.uol.edu.pk https://sites.google.com/site/pctresearchgroup/

2. Outline Introduction ATLAS Project ATLAS corpus and lexicon The LIS Dictionary Architecture The ATLAS Visualization system Web and Mobile Application Conclusion and Future Work

3. Introduction Sign language is a gestural visual language with signs as lexical units instead of words. Sign languages have their own grammars, which are quite different from oral languages. [1] Sign Language is not a universal language; each country has its own Sign Language American Sign Language (ASL) British Sign Language (BSL) Italian Sign Language (LIS) In some countries more than one Sign Language In Switzerland, Swiss-German, Swiss-French and Swiss- Italian Sign Language(s) are used. [2] 1.Liddell, S.K. Grammar, gesture and meaning in American Sign Language, Cambridge Univ Pr (2003) 2.Timmermans, N. The status of sign languages in Europe, Council of Europe, 2005

4. Introduction We proposed Platform Independent Lingua dei Segni Italiana (LIS) Dictionary to support learning of Sign Language and written language LIS Dictionary provides full set of lexemes and videos data set of Italian sign language with an online interface attached to the MultiWordNet. By linking LIS lemmas to the corresponding closest concepts in the MultiWordNet lexical database enhances the possibility to understand the new words from existing signs.

5. ATLAS Project Targets automatic translation of written contents Defines a web platform and a service Requires the collection of LIS resources Output provided by means of a virtual character Provides several software applications Translator On- line dictionaries An editor for the manual translation from Italian to LIS The signed content is provided on several platforms Digital Terrestrial Television Word Wide Web Mobile

6. ATLAS Partners The project collaborates with companies and academic institutions selected according to their characteristics of excellence in specific areas Polytechnic of Turin, Italy University of Turin, Italy University of Illinois at Chicago Italian RAI SpA Microsoft Innovation Center BEPS Engineering Lumiq Studios Ltd Virtual Reality and Multimedia Park and some others

7. ATLAS Objective The main objective of ATLAS project is to develop tools that allow naturally challenged people to follow and understand the contents of television, multimedia and entertainment. More specifically, ATLAS aims at developing a virtual interpreter, which automatically translates Italian into LIS.

8. Usage To load LIS visualization service on the cell phone or PDA for the deaf traveller to understand announcements on Railway stations and airports.

9. VIRTUAL ACTOR COMMANDS TEXT VIDEO TEXT Virtual Actor Commands SOURCES Teletext Subtitles Audio Audio/Video SOURCES Teletext Subtitles Audio Audio/Video Transformations TransformationsTransformations AEWLIS Sentences AEWLIS Sentences ATLAS Work Flow

10. ATLAS corpus and lexicon The Platform Independent LIS Dictionary is part of the Automatic Translation into Sign Languages (ATLAS) Project. As a significant amount of parallel data is not available yet, we have started building a new corpus from scratch. Our corpus contain weather forecast bulletins broadcasted by Italian National TV Channel (RAI TV) for a total of about 15K Italian running words and about 1.5 hours of LIS videos We have adopted mixed approach for the generation of signs, using an animation language to compose and parameterize pre-captured and hand animated signs.

11. ATLAS corpus and lexicon Our Sign Language dictionary provides more than three thousand sign videos. These sign videos are set of Avatar animations which are rendered through a graphic engine. Each sign is linked to a corresponding Italian lemma existing in the MultiwordNet. About twenty seven hundred video animations were created using standard Lingua dei Segni Italiana dictionary. More than four hundred signs are custom created by a team of expert signers and deaf.

12. The LIS Dictionary Architecture The Sign Language Dictionary takes Italian lemmas as input and provides as output their representation in the Italian Sign Language. The process for the generation of avatar based signs in the LIS Dictionary is as follows.

13. The ATLAS Visualization system The module used for the signs generation is system that generates real-time animations, display them in a player and export them as picture frames. It takes as input the symbolic form of LIS called ATLAS Written Extended LIS (AEWLIS) [3], a sequence of lemmata and a description of the meaning, syntactic number and other semantic, pragmatic and articulatory information. Signs are described by a default animation (stored in a repository, the “signary”), which has to be parameterized with respect to the context given by the specific sentence. 3.N. Bertoldi, G. Tiotto, P. Prinetto, E. Piccolo, F. Nunnari, V.Lombardo, R. Damiano, L.Lesmo, and A. D. Principe, “On the creation and annotation of a large-scale italian-Lis parallel corpus,” in 4th Workshop on the Representation and Processing of Sign Languages:Corpora and Sign language Technologies (CSLT 2010), 2010.

14. The LIS Dictionary Architecture

15. The Italian Sign Language Dictionary Provides more than 3k signs Signs stored as set of animations Provides link with Italian written lemmas Meanings in written Italian Links with MultiWordNet

16. Italian Sign Language Dictionary

17. Italian Sign Language Dictionary Coverage

19. Sign Language Mobile Dictionary

23. Conclusion and Future Work We discussed a novel platform independent Italian Sign Language dictionary, which is capable to run on several devices and is linked with MultiWordNet. It is available in the form Web application online and as a mobile application for Windows Phone7, iPhone and Android. Our dictionary contains a large video data set and its structure is flexible enough to include further signs in future. We plan to develop web applications that can help in learning of new words. The objective is to propose web based interface as a tool to enhance deaf learning of written Italian Language in schools.

24. Conclusion and Future Work Different scenarios of LIS vocabulary will be created to check the coherence and understanding and hence get feedback on lexical resources by deaf users with help of some expert signers. The subjects would be set of deaf users, normal people having different literacy level and aptitude. The community feedback will be helpful to check the originality of the material and comprehensiveness of lexical resources. Since MultiWordNet has multilingual lexicons, such as Spanish, German and Portuguese, it could be helpful for written multi-language learning. ATLAS LIS WEB APPS

25. Questions?