1. CPSC 643 Aligning Windows of Live Video from an Imprecise Pan-Tilt-Zoom Robotic Camera into a Remote Panoramic Display Dezhen Song Department of Computer Science and Engineering Texas A&M University Supported in part by

2. 2 Network PTZ Robotic Camera for Nature Observation Panosonic HCM 280 –PTZ Robotic Camera: 350° Pan, 120° Tilt, 42x Zoom 200° per second servo speed –Network Video Camera: Built-in streaming server 640x480 pixels video >30 frames per second –Low power consumption: <5 Watt –Affordable price: $ 1.2 K

3. 3 Real Time Panoramic Video Tilt Pan Frame sequence Panorama Tilt Time Panorama Live frame sequence Updated Part in Panorama

4. 4

5. 5 Related Work –Multiple fixed cameras [Swaminathan and Nayar 2000] [Tan et al. 2004] [Foote et al. 2000, 2001] –Single wide angle camera [Baker and Nayar 1999] [Nayar 1997] [Xing and Turkowski 1997]

6. 6 Related Work: Image Alignment –Direct Method Use pixel intensity value Sensitive to luminance change Need good guess for initial parameters input Existing work –[Shum and Szeliski 1997] [Szeliski 1994, 1996] –[Coorg and Teller 2000] [Kang and Weiss 1997] –Frequency Domain Registration Existing work –[Castro and Morandi 1987] [Reddy and Chatterji 1996]

7. 7 Related Work: Image Alignment –Feature-based Image Registration Use feature points: Harris corner point, SIFT Robust to luminance change Faster than direct method Existing work –[Torr and Zisserman 1997] [Brown and Lowe 2003] –[Zoghlami et al. 1997] [Hu et al. 2001] [Cho et al. 2003] –[Kanazawa and Kanatani 2002] [Zhang et al. 2002]

8. 8 Comparison: Panoramic Video SystemResolutionBandwidthLive Motion Images Our systemExcellentLowYes Film-based panorama ExcellentLowNo Wide-angle systems PoorModerateYes Multi-camerasGoodModerate to High Yes

9. 9 Assumptions Pan-tilt camera with a fixed base Known intrinsic camera parameters –Calibrated camera before deployment Inaccurate pan-tilt readings –May deteriorate over time Standard video camera with HFOV ≤ 45 o

10. 10 Review: Perspective Projection Intrinsic ParametersExtrinsic Parameters [Tsai86, 87]

11. 11 Problem Definition Re-projection: Project image B onto image A plane: Image alignment:

12. 12 Excessive Computation in Image Alignment Speed slow down caused by coupling re-projection and SSD: –Extensive float point computation –Coupled with Sum of Squared Difference (SSD) operation –A naive search takes O(km) re-projection operations k: number of candidate pan/tilt pairs over feasible solution set. m: number of overlapping pixels Proposed solution: decouple re-projection and SSD –Spherical re-projection –Cell-based Alignment Constant time alignment

13. 13 Project image onto a spherical surface Image =(p, t) on local spherical coordinate system { } Spherical Projection CYCY CXCX t p u v O Image plane f CZCZ

14. 14 0.8 0.4 -0.4 -0.8 -1.5-0.500.5 p t 0 -400 -200 0 200 400 v 200 u -200-600-1000 PlanarSpherical Two poses have 30 o pan difference with the same 30 o tilt value Distortion under Re-projection

15. 15 0.8 0.4 -0.4 -0.8 -1.5-0.500.5 p t 0 Invariant under Spherical Re-projection

16. 16 Re-projection after Spherical Projection Define conversion between camera coordinate system and local spherical coordinate system Re-projection function between two local spherical coordinate system

17. 17 R c is a 2x2 rotation matrix Cell distortion under re-projection is negligible. Lemma 1 : If the spherical cell is small, define point and its corresponding point we have

18. 18 Proof for Lemma 1: Introduce coefficient matrix H Radius f remains same

19. 19 Continue: Proof for Lemma 1 HFOV≤45 o and VFOV ≤34 o  0.956 ≤ cos(t) ≤ 1 Dropping cos(t) introduces ≤ 5% distortion for 20x20 cell ∆f=0  substitute [m 13, m 23, m 33 ] T

20. 20 Continue: Proof for Lemma 1

21. 21 Lemma 2: Rotation angle Θ of R c can be approximated by where α is the dot product of Z axis of {C A } and {C B }

22. 22 Algorithm CjCj ε( C j )

23. 23 Cell based Image Alignment Select k c cells from the overlapping region inO(1) Sphere projectionO(1) Feature detection in the cell and searching regionsO(1) For each(δp B, δt B )O(1) For each cellO(1) Compute C j Compute, j=1, …, k c Compute SSD between and End For Report sum of SSD across all cells End For Output solution with the minimum SSD

24. 24 Experiment and Results Speed test: –881 milliseconds to align 21 320x240 images –4 seconds for Autostitch program on same data set –Up to 25fps on a laptop PC

25. 25

26. 26 Align video images from a camera that only differ in pan and tilt settings into a panorama at 25 frames per second. Alignment is performed on a spherical surface to avoid excessive distortion caused by homographic transformation. A constant time algorithm pre-rotates small pre-sampled squared patches on spherical surface for matching. Experiments show that the alignment speed is 4.5x faster than the best method available. live video window Thank You!