1 Multimedia Information and Mobile-Learning Duc Phuong Nguyen Martin Guggisberg Helmar Burkhart University of Basel Switzerland

2 Challenge 1: Content adaptation Jobs done: 40 Queues : 30 Status: OK Elapsed time: 1 hour Remain time: 2 hours Short text can be sent via SMS Client-side globus-hostname command is not returning a fully qualified domain name (FQN). The submission scripts use globus-hostname to send information to the server about how to contact the client. The fix is to make globus-hostname return the FQDN. There are firewalls and ports (GRAM: 2119, LDAP: 2135) which are not open. Check that GLOBUS_TCP_PORT_RANGE is set to 3000 to 3090 on the client (or another range acceptable for the server to contact the client on), and that all intermediate firewalls are open for this range in both directions. GRAM Job submission failed because the executable file permissions do not allow execution". Probable cause: You submitted a script, e.g., globus-job-run symphony:2119/jobmanager-loadleveler $PWD/simple. ll but simple.ll does not have the execute permission bit set. The error msg "Authentication Failed remote certificate signed by unknown CA" means the client cannot find the CA for the cert presented by the server. Make sure the CA cert is present in the trusted cert dir set with setup-gsi. Also, check the time is consistent across the machines. The error msg "GRAM Job submission failed because the connection to the server failed (check host and port) Make sure that the gatekeeper service is enabled. The error msg "GRAM Job submission failed because data transfer to the server failed" means the job manager has problems opening some file. Make sure $GLOBUS_LOCATION is readable by all the users (mode 755) What about this text? How can we sent this information to mobile users?

3 The second message started on at 17h … ………………………………………………… Challenge 2: Content overview Difficult to get the most important message first: „You have 10 voice messages in your mail box…The first message started on at 14h … ………………………………………………… The ninth message started on at 17h … ………………………………………………… The last message is actually what important to me in this example“

4 Challenge 3: Integration Suppose that we have a system that solves Challenge 1 (content adaptation) Challenge 2 (content overview) How can we integrate it into our E-Learning system?

5 Solution 1: Content adaptation CoMobile Speech1.wav Speech2.wav Long texts Audio Voice call Quite difficult to convert mathematic formulae to speech

6 Solution 2: Content overview CoMobile Keywords extraction (Post processing) Audio 1. Lunch 2. Food 3. Project meeting 4. Concert …………….. Speech1.wav Speech2.wav Voice call Recorded lectures Text List of keyword can be used to identify the important voice messages SMS

7 CoMobile framework- Uni Basel - Access learning materials - Search, notification - Collaboration (sharing, discussion) - Self-assessment, voting, quiz

8 CoMobile framework- Uni Basel - Access learning materials - Search, notification - Collaboration (sharing, discussion) - Review, self-assessment, voting, quiz

9 CoMobile-Example CoMobile Voice message Alice Voice message Bob Text message Christian Voice and Text Pengi

10 CoMobile-Example CoMobile Voice message Voice and Text Text message

11 CoMobile framework-Overview 1. Content adaptation: long texts are converted into speech and then can be accessed by mobile phones 2. Search on audio data 3. Platforms and languages independence 4. Flexible: easy to integrate via CoMobile-services

12 CoMobile framework-Components (I) All components are open source software: - CoMobile main component - Plug-in components: - Text to Speech: Festival - Speech recognition: Sphinx - Optical character recognition: GOCR Plug-in components can be replaced without many modifications.

13 CoMobile framework-Components (II) Service-oriented: works on document‘s level (e.g: input=audio files, output=text files) CoMobile calls the component‘s public service The component invokes CoMobile callback service when it finish the task Communication between CoMobile and the plug-in components are done via HTTP

14 CoMobile framework- Easy to integrate Service-oriented simplifies the intergration process: PHP-based system with CoMobileJava-based system with CoMobile

15 CoMobile framework-Components (III) Festival-Text to Speech: support many languages (English, German, French, Spanish, etc.) Sphinx-Speech recognition: free acoustic model for English ( words; WER = 18%) Optical Character Recognition: GOCR, tesseract (supported by Google). Recognition rate is still quite low for handwritten notes (accuracy rate= appr. 20% with empirical tests).

16 Open issue: Recognition of notes taken by mobile phone camera Handwritten notes and the low resolution of mobile phone‘s camera decrease the accuracy of OCR systems significantly

17 CoMobile example: Existing Web-based applications can use mobile phones to access and submit information by integrating with CoMobile services Text retrieval, features extraction, vector space retrieval, inverted list, ranking

18 Uni Basel-CoMobile features Content adaptation and content overview for mobile devices. Data submission: send information (text, images, audio, video) via SMS, MMS, voice recording. SMS notification service (replies, reminds, alerts). Access information via various channels: Voice, SMS, MMS, WAP, Bluetooth.

19 Transformation issue Information loss in transformation chain Recognition accuracy Quality of search: keywords, semantic TextAudio Graphic

20 Conclusion Content transformation is an important issues in mobile applications. Uni Basel-CoMobile: an easy to integrate and flexible framework that support collaborative learning with mobile devices. The CoMobile framework is open source and look forward for contributions.

21 Uni Basel-CoMobile-Demo CoMobile demonstration

22 Discussion Cost: Users have to pay for SMS, MMS, as well as voice call. „Flat rate“ package will be common in the near future. Storage: 1 minute video = 1 MB (approximately). There exists 2GB flash card for mobile phone Bandwidth: 14.4 kbps (GSM), 140 kbps (GPRS), 1920 kbps (UMTS)

23 SMS gateway - Kannel RS-232 AT Commands AT+CNMI=1,2,0,1,1 SMS Text HTTP request to forward SMS Text

24 MMS gateway - Mbuni RS-232 AT Commands AT+CNMI=1,2,0,1,1 MMS Text, Picture, Audio HTTP request to forward MMS Mbuni

25 PBX Asterisk PCI socket Voice AGI: Perl, Java, PHP scripts are called Telephone line