Initiation of Standardization on Network-based Speech-to-speech Translation at ITU-T SG16 National Institute of Information and Communications Technology, Japan Satoshi Nakamura Chiori Hori Contact :Name Satoshi Nakamura Chiori Hori Organization NICT Country Japan Tel: Fax: INTERNATIONAL TELECOMMUNICATION COM 16 – C 196 – E TELECOMMUNICATION STANDARDIZATION SECTOR STUDY PERIOD October 2009 English only Original: English Question(s):7, 21, 22/16

Many Languages All Over the World

Breaking Language Boundaries  Language boundaries is one of the causes of barriers to mutual understanding.  To remove language boundaries between people who speak different languages, Speech-to-Speech Translation (S2ST) technologies are an effective means of communication.  S2ST technologies have been studied.

English “I go to school” SpeechRecognition(ASR) MachineTranslation(MT) SpeechSynthesis(TTS) w a t a sh i w a g a xtu k o o n i….. 私は学校に行く I to school go I go to school Japanese 「私は学校に行 く」 Corpora Convert to English word sequence 「私は」⇒ “I” 「学校に」⇒ “to school” 「行く」⇒ “go” Convert to word sequence using lexicon and grammar Convert to Japanese phoneme sequence “w”, “a”, “t”… Select appropriate waveform for English text Reorder word sequences according to English grammar “I” “I” “to school” “go” “go” “to school” Speech-to-Speech Translation (S2ST)

Stand Alone and Client-server S2ST Systems Stand alone system Japanese English Chinese Indonesian Packages the entire speech translation functions into a handheld PC Japanese speech “ おはようございま す． ” English speech “Good morning.” Client-server system

Why Network-based?  Resource limitation in stand alone systems and language pairs are limited.  ASR/MT/TTS systems for many languages are available and needs to be maintained by each country.  Broadband network is available.

Standardization on Network-based S2ST Language B ASR MT Language A Language B Parallel corpus lexicon Lexicon Speech data Language A Lexicon Speech data Language A Lexicon Speech data Language B Lexicon Speech data TTS Parallel corpus lexicon Speech of Language B Speech of Language A Synthesized Speech Parallel corpus, Speech data, lexicon Standardization Parallel corpus, Speech data, lexicon Standardization Data format for ASR and MT results Data format for ASR and MT results Communication protocol among modules TTS S2ST Client MT ASR Synthesized Speech S2ST Client

Lexicon for overall S2ST systems An example of a lexicon for overall modules in S2ST systems  The global standardization for lexicon format and a system to collect and provide lexicon for all languages is requisite to maintaining reliable lexicon for overall S2ST systems.

Asian Network-Based S2ST System by A-STAR Consortium 1 National Institute of Information and Communications Technology (NICT), Japan 2 Electronics and Telecommunications Research Institute (ETRI), Korea 3 Chinese Academy of Sciences (CASIA), China 4 National Electronics and Computer Technology Center (NECTEC), Thailand 5 Agency for the Assessment and Application of Technology (BPPT), Indonesia 6 Center for Development of Advance Computing (CDAC), India 7 Institute of Information Technology (IOIT), Vietnam 8 Institute for Infocomm Research (I2R), Singapore

Server Location for Network-based S2ST

Speech Translation using Distributed Service Servers Example: From Korean to Thai Speech Translation Speech translation service client TTSserver ASRserver ① Speech recognition (Korean) ② Language translation (Korean→Thai) Synthesized speech (Thai) MTserver Translated text (Thai) Speech (Korean) MTserver TTSserver Text (Korean) ASRserver ③ Speech synthesis ( Thai)

S2ST Client and Server 12

Scope of Standardization DraftTitleScopeTarget Date F.S2STreqsFunctional Requirements for Network- based S2ST  Definition of network-based S2ST  Functions and service requirements of network-based S2ST During this Study Period ( ) H.S2STarchArchitectural Requirements for Network- based S2ST  Functional architectures, mechanisms and  interface of network-based S2ST During this Study Period ( ) Table : Draft Roadmap to develop standards for network-based S2ST

Conclusion  We would like to invite more people to standardization activities on network-based S2ST systems.  By leveraging the standardization, network-based S2ST systems can cover more languages.