On-Demand Sharing of a High-Resolution Panorama Video from Networked Robotic Cameras Supported in part by CPSC 643 Dezhen Song Texas A&M University

2 Panosonic HCM 280 –PTZ Robotic Camera: 350° Pan, 120° Tilt, 42x Zoom Maximum spatial resolution: 500 Megapixel per steradian 3 Gigapixels panorama –Network Video Camera: Built-in streaming server 640x480 pixels video >30 frames per second Network PTZ Robotic Camera for Nature Observation

3

4

5 Giga-pixel Motion Panorama VS. Fixed Lens Camera Fixed lens with mirror 10M Pixel CCD $ 20.0 K 2M Pixel / Steradian Pan, Tilt, Zoom (21x) 0.37M Pixel CCD $ 1.2 K 500M Pixel / Steradian

6 Existing Panoramic Video Systems SystemCamera Band width Video Output Sample Systems Wide angle lens/mirrors Single fixed Low Low quality live stream [Baker 1999], [Nayar 1997], [Xiong 1997], [Ng 2005] Multiple camera panorama video Multiple fixed High Live panoramic video [Foote2001], [Swaminathan 2000] Panoramic video texture Single pan High Pseudo-live panorama video by changing video temporal display [Agarwala 2005] Dynamosaics Single pan High Pseudo-live panorama video by changing space-time volume [Rav2005] Motion panorama SingleLow Static panorama overlaid with living moving objects trajectory [Irani 1996], [Bartoli 2004] Our system PTZ Cameras Low Partial live panorama

7 Evolving Panorama: High Resolution Live Panoramic Video Using PTZ Camera Tilt Pan Frame sequence Panorama Tilt Time Panorama Live frame sequence Updated Part in Panorama

8 robotic video cameras Collaborative Observatories for Natural Environments ( motion sensors timed checks sensor network s humans: amateurs and profs

9 On-demand Panoramic Video Sharing

10 Challenges: –Dynamic video coverage –High resolution panorama coverage –Multiple different spatial-temporal client requests. On-demand Panoramic Video Sharing

11 User Request Live Time k-1 k … … … Live video … Client i User i request: r i =[u, v, w, h, t s, t e ]

12 Camera Coverage pan -180 o 60 o tilt N 1 p jk Patch-based Panorama Video Snapshot at time k Live patchStatic patch Camera Coverage

13 Patch-based Panorama Video Live video Live Time k-1k … … … k-2 … Camera coverage at time k Patch j at time k

14 On-demand Patch-based Panorama Video Sharing

15 Frame Insertion Algorithm Input: F t Output: Updated evolving panorama video Wrap F t onto the spherical surface; Estimate F t ’s registration parameters by aligning it with previous frames; Project F t onto the sphere panorama surface; for each p j and p j ∩ F t ≠ Ø do Insert p jt into p j ’ s GOP buffer; for each p j, j=1, …,N do if p j ’s GOP buffer is full then Encode patch video segment; Store patch segment start position and time data into lookup table; Reset GOP buffer for incoming data;

16 On-demand Patch-based Panorama Video Sharing For User i request: Send patch data: r i =[u, v, w, h, t s, t e ] r i ∩ P t = { p jk | j Є {1,…,N}, k Є [t s, t e ], p jk ∩ r i ≠Ø, p jk ≠Ø }

17 User Query Algorithm Input: r i Output: r i ∩ P in MPEG-2 format Identify patch set S= { p j | j Є { 1,…,N }, p j ∩ r i ≠Ø }; for each p j Є S do Find the nearest I frame p jb earlier or equal to t s ; Find the nearest I frame p jc later or equal to t e ; Transmit the patch segments between p jb and p jc ;

18 Experiments and Results Hardware configuration : Dell Dimension DX, 3.2Ghz Pentium dual-core processor, 2GB RAM Panasonic HCM 280A video camera Software configuration : Visual C++ in Microsoft Visual Studio 2003.NET MPEG-2 encoder/decoder from MPEG Software Simulation Group Input data set : Frame number: 609 Frame resolution: 640x480 pixels Frame rate: 25 fps Raw RGB data size; 536 MB Panorama resolution: 2742x909 pixels

19 Experiments and Results Storage and computation speed versus different patch sizes:

20 Experiments and Results Bandwidth for a user query (800x600 pixel) versus different patch sizes:

21 Summary Patch-based data representation and encoder provides on-demand sharing of a high resolution panoramic video from networked robotic Pan-Tilt-Zoom cameras with: Effective data organization Efficient data storage. Satisfy spatial-temporal user video requests.