Design and Implementation of a Wireless Handheld Multimodal Digital Video Library Client System Sam K. S. Sze Henry K. P. Choi Feb Group Meeting

Abstract We have developed a client system for accessing a multimodal digital video library (DVL) namely, iVIEW. It provides a user interface that meets the challenge of rich multimodal information presentation on wireless handheld device. An XML schema is employed to organize the multimodal metadata. Furthermore, we investigated a context awareness mechanism complementary to the XML schema to facilitate scalable degradation under restricted resources in wireless application environment.

Presentation Overview Introduction iVIEW Overview – A Practical Perspective iVIEW Wireless Client Scalable Degradation Conclusion

Introduction Evolving DVL technologies Common challenges for wireless handheld device application development Challenges for implementing wireless digital video library client system

Introduction: Evolving DVL Technologies Multimodal information extraction and organization. The modals includes: speech, on-screen characters, geographical locations, names, faces, on- screen objects … Internet access Wireless access at anytime and anywhere

Introduction: Common Challenges of Wireless Handheld Devices Limitations Inherited from Hardware Screen Size Input Device (Pen device) CPU Power Memory Battery Life Bandwidth Limitations

Introduction: Challenges for Implementing a Wireless Digital Video Library Client Intelligent User Interface Content Selection Result Set Refinement Integrate Handheld Platform Operation Manner Resources Management

iVIEW Overview

iVIEW Overview: If Someone Comes to Room 121 and Ask VIEW Technologies …. I want my video content to be searchable over Internet and wireless tomorrow. What kind of stuff can you guys provide? ?

iVIEW Overview: What’s your answer? My Answer is … Demonstration

iVIEW Client Architecture

iVIEW Client: iVIEW Client System Development The iVIEW Client has been developed in several approaches: As a Microsoft Windows Native Application using Visual Basic as the frontend with a number of Visual C++ DLLs. (CMU version, Michael Christel) Further extended to support both English and Chinese. (April 2001, Sam) As a Java Applet for internet access using internal frames (Jacky FYP). Extended to support Real and Quicktime streaming, XML message parsing (August 2001, Sunny, Sam). As a web page for internet access using Dynamic HTML and ASP. (December 2001, Sam). Wireless …

iVIEW Client: A History of iVIEW Wireless Client April 2001 (1 month) Survey - Sam June 2001, (3 weeks) Wireless Web Version – Gordon, Sam August 2001, (2 weeks) Framework of Multi-windows native application using Embedded Visual C++ - Sam October 2001 (2 weeks) Bluetooth and GSM compliance tested – Sam, Mike November 2001 (3 weeks) A variant version with HTML viewer control – Mike December 2001 (1 month) A multilingual search engine using ASP – Sam, Gordon January 2002 (1 month) A full version of the multi-windows application - Henry February 2002 (2 weeks) A performance monitor added - Henry

iVIEW Wireless Client: Implementation Platform: iPAQ PocketPC Wireless connections: GSM HSCSD 2.5G CDMA b Bluetooth Multi-windows User Interface 3 Phrases of Operation: Query Result Set Manipulation Presentation

iVIEW Wireless Client: Query Chinese and English Query

iVIEW Wireless Client: Result Set Manipulations Segment View, Tree View, Topics View

iVIEW Wireless Client: Presentation GOVERNMENT HAS RESTORED FULL DIPLOMATIC RELATIONS WITH LIBYA AFTER A 15 YEAR SUSPENSION. CNN'S MARGARET LOWRIE HAS THE DETAILS FROM LONDON LONDON BRITISH FOREIGN SECRETARY ROBIN COOK ANNOUNCED DIPLOMATIC TIES WITH LIBYA WOULD BE RESTORED BECAUSE IT IT NOW ACCEPTS RESPONSIBILITY.....

iVIEW Wireless Client: Gracefully Degradation at Multimodal Presentation

iVIEW Wireless Client: Embedded Resources Monitor Resources CPU Usage Memory Utilization Bandwidth Consumption

iVIEW Wireless Client: Gracefully Degradation at Multimodal Presentation A set modal significance order chain (i.e. S(m 1 ) > S(m 2 ) > S(m 3 )> …) should also be defined for different video types. If system utilization saturates, the dispatch will be signaled to make a decision to stop dispatching a set of active modal of least significance. Mathematically,

iVIEW Wireless Client: Measurement – A Typical Client Operation Profile

iVIEW Wireless Client: Measurement – Degrade Transition

Conclusion A wireless DVL client system, that meets the challenges of handling complex multimodal presentations in a handheld device, had been developed. We had shown a user interface design that allow flexible content selection, result set refinement in different views that fits the pen computing operation profile, and it also equips the capability of high complexity multimodal presentation. We define a multimodal presentation XML with its corresponding context aware scalable degrade mechanism that optimizes the presentation content under resources limited and fluctuating environment.

Q & A