Sub-Project 2: Multimedia Distribution Platforms for Cyber-Physical Systems NUS PI: Roger Zimmermann ZJU PI: TBD NUS Co-PIs: Ben Leong ZJU Co-PIs: TBD 11/27/2018 1

Motivation (1) Challenge Current large-scale multimedia systems, such as networked games, are mostly built based on an integrated digital media distribution architecture that is designed from scratch for each application. Goal Aims to design a general-purpose, extensible common platform that addresses the requirements of all networked CPS apps.For example: Media transmission protocols Sensor- and meta-data annotations … 11/27/2018 2

Motivation (2) Trends User-generated video content is growing rapidly. Mobile devices make it easy to capture video. Example Mobile Video Scenario (GeoVid) Viewable scene modeling (i.e., geo-properties) Video and (sensor) meta-data acquisition Indexing, querying, and presentation of results 1) 2) 3) d  11/27/2018 3

Ex.: Spato-Temporal Range Query on Video Streams area 11/27/2018 4

Search and Result Presentation (2D) http://geovid.org 11/27/2018 5

GeoVid Demonstration

Architecture of Mobile Sensor-Rich Video Management System User Interface Meta-data Representation Meta-data Meta-data Processor Viewable scene descriptions Mobile Device User Missing Video frame request Indexing and Storage Spatio-temporal range query Query Interpreter Query Video content Video Ranking Video results Spatio-temporal Relevance User Spatio-temporal overlap properties Video Summarization 11/27/2018 7

Dynamic Adaptive Streaming over HTTP (DASH) Traditional media streaming (e.g., based on RTP/RTSP/RTCP) streaming faces several challenges in large-scale systems: Special-purpose servers/proxies/peers for media (complex) Difficulties in firewall traversal: protocols use both TCP and UDP transmissions Difficulties in caching data (no “web caching”) Advantage of RTP/RTSP/RTCP Short end-to-end latency 11/27/2018 8

DASH Standard (Dec. 2011) ISO/IEC Standard: “Information technology — MPEG systems technologies — Part 6: Dynamic adaptive streaming over HTTP (DASH)” JTC 1/SC 29; FCD 23001-6 Uses HTTP protocol to “stream” media Divides media into small chunks, i.e., streamlets Makes playback adaptive Encode media into multiple different streamlet files, e.g., a low, medium, and high quality version (different bandwidth). Supported by: Move Networks, Apple, Microsoft, Netflix, … 11/27/2018 9

DASH Limitations DASH standard only describes server-to-client dissemination: Upload mechanism is undefined! We are working on a low latency DASH upload protocol for mobile devices. MPD: Media Presentation Description 11/27/2018 10

DASH Upload Design Features: Real-time segmentation on the mobile device Chunked, resumable upload via HTTP Segment-level, server-side transcoding for quality adaptation [ACM MMSys 2012] End-to-end startup latency on iPhone 4: 11/27/2018 11

Smartphone Apps Android App http://geovid.org/Android/index.html Available for download iPhone App http://geovid.org/iphone/index.html Available in the iTunes App Store 11/27/2018 12

Search and Result Presentation (3D) e.g., with Google Earth 11/27/2018 13

Other Research Peer-to-peer video and audio streaming Texture streaming in virtual worlds Geo-graphic routing and sensor networks Mesh networking VoIP and HD communications Etc.

Multimedia Distribution Platforms for Cyber-Physical Systems Core deliverables The cyber-physical node server that would effectively be the web-server equivalent for the cyber-physical web. The hyper-renderer client for a cyber-physical environment server - equivalent of the normal web-browser for the cyber-physical web server. Co-space, which is the whole system, comprising of the servers, middleware, and the clients, will then be used to develop the various cyber-physical applications. 11/27/2018 15

Thank You http://geovid.org Roger Zimmermann, Seon Ho Kim, Ben Leong Sakire Arslan Ay, Beomjoo Seo, Min Min Htoon, Liang Ke, Hao Jia, Wang Guanfeng, Ma He, Fang Shunkai, Zhang Lingyan, Shen Zhijie, Zhang Ying, Cui Weiwei, Yin Yifang National University of Singapore University of Southern California Further information at: http://geovid.org rogerz@comp.nus.edu.sg seonkim@usc.edu 11/27/2018 16