1 Frame Selection Algorithms for Collaboratively Tele-Operated Robotic Cameras

2 Conventional robotic webcamera Collaboratively controlled robotic webcamera

3

4

5 Frame Selection Problem: Given n requests, find optimal frame One Optimal Frame

6 Requested Viewing Zones Optimal Satellite Frame Oct. 27, 2003

7 Satellite Imaging 2.44 Billion Market in 2001 Increasing 14% per year since 1999 Major clients –Government / Military –Oil Exploration –Weather Prediction –Agriculture Ikonos, 1999

8 Related Work Facility Location Problems –Megiddo and Supowit [84] –Eppstein [97] –Halperin et al. [02] Rectangle Fitting, Range Search, Range Sum, and Dominance Sum –Friesen and Chan [93] –Kapelio et al [95] –Mount et al [96] –Grossi and Italiano [99,00] –Agarwal and Erickson [99] –Zhang [02]

9 Related Work Similarity Measures –Kavraki [98] –Broder et al [98, 00] –Veltkamp and Hagedoorn [00] CSCW, Multimedia –Baecker [92], Meyers [96] –Kuzuoka et al [00] –Gasser [00], Hayes et al [01] –Shipman [99], Kerne [03], Li [01]

10 Problem Definition Assumptions –Camera has fixed aspect ratio: 4 x 3 –Candidate frame c = [x, y, z] t –(x, y)  R 2 (continuous set) – Resolution z  Z Z = 10 means a pixel in the image = 10×10m 2 area Bigger z = larger frame = lower resolution (x, y) 3z 4z

11 Problem Definition Requests : r i =[x l i, y t i, x r i, y b i, z i ], i=1,…,n ( x l i, y t i ) ( x r i, y b i )

12 Optimization Problem User i’s satisfaction Total satisfaction

13 Problem Definition “Satisfaction” for user i: 0  S i  1 S i = 0 S i = 1  = c  r i c = r i

14 Measure user i’s satisfaction: Coverage-Resolution Ratio Metrics Requested frame r i Area= a i Candidate frame c Area = a pipi

15 Comparison with Similarity Metrics Symmetric Difference Intersection-Over-Union Nonlinear functions of (x,y), Does not measure resolution difference

16 Optimization Problem

17 Co-Opticon Problem Versions Fixed Resolution Exact Algorithm Variable Resolution Exact Algorithm Approximate Algorithm for Arbitrarily- Shaped Requested Frame Distributed Algorithms D. Song, A.F. van der Stappen, and K. Goldberg, Exact and Distributed Algorithms for Collaborative Camera Control, the Fifth International Workshop on Algorithmic Foundations of Robotics. Nice, France, Dec 15~17, 2002.

18 Requested Frame r i Candidate Frame c (for fixed z) Objective Function Properties

19 s i (x,y) is a plateau One top plane Four side planes Quadratic surfaces at corners Critical boundaries: 4 horizontal, 4 vertical Objective Function for Fixed Resolution 4z x y 3z 4(z i -z)

20 Objective Function Total satisfaction: for fixed z Frame selection problem: Find c * = arg max S(c)

21 S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines. Objective Function Properties 4z x y 3z 4(z i -z) 3z y sisi (z/z i ) 2 3(z i -z) x sisi 4z (z/z i ) 2 4(z i -z)

22 Plateau Vertex Definition Intersection between boundaries –Self intersection: –Plateau intersection : y x

23 Plateau Vertex Optimality Condition Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof: –Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur at x boundaries y S(y)S(y)

24 Fixed Resolution Exact Algorithm Brute force Exact Algorithm: Check all plateau vertices  (n 2 ) plateau vertices  (n) time to evaluate S for each  (n 3 ) total runtime

25 Improved Fixed Resolution Algorithm Sweep horizontally: solve at each vertical –Sort critical points along y axis: O(n log n) –1D problem at each vertical boundary O(n) –O(n) 1D problems –O(n 2 ) total runtime O(n) 1D problems y S(y)S(y) x y

26 Speed comparison Random inputs Curve B: Brute force approach Curve V: using line sweeping

27 More Improvements for Fixed Resolution Har-Peled, Koltun, Song, and Goldberg. [03] –Exact algorithm O(n 3/2 log 3 n) –Near Linear Approximation Algorithm O(NlogN) N = O(nE) E = (log(1/ε)/ε) 2, where ε is the approximation bound S. Har-Peled, V. Koltun, D. Song, and K. Goldberg, Efficient Algorithms for Shared Camera Control, In Proceedings of the 19th ACM Symposium on Computational Geometry, 2003.

28 Co-Opticon Problem Versions Fixed Resolution Exact Algorithm Variable Resolution Exact Algorithm Approximate Algorithm for Arbitrarily- Shaped Requested Frame Distributed Algorithms Dezhen Song, A. Frank van der Stappen, and Ken Goldberg, An Exact Algorithm Optimizing Coverage-Resolution for Automated Satellite Frame Selection, (To appear) IEEE International Conference on Robotics and Automation (ICRA) 2004

29 Virtual Corner A two-requested frame case –Requested frame: y x

30 Virtual Corner Virtual corner definition –Real corner: –Extended edge intersections : y x

31 Recall: Plateau Vertex Definition Intersection between boundaries –Self intersection: –Plateau intersection : y x

32 Virtual Corner and Plateau Vertex Intersection between boundaries –Candidate frame: –Frame intersection : y x

33 Virtual Corner and Plateau Vertex Intersection between boundaries –Candidate frame: –Virtual corner : y x

34 Variable Resolution Exact Algorithm Lemma: At least one optimal frame has its corner overlapped with virtual corner. –O(n 2 ) Virtual corners –One 3D problem→ O(n 2 ) 1D sub problems r6r6 r2r2 r5r5 r3r3 x y r4r4 r1r1 O z S(z) Candidate frame

35 Overall complexity –O(n 2 ) 1D problems –O(n) sub 1D problems –O(n) to compute polynomial coefficient for each sub 1D problem s(z) = g 0 z -1 +g 1 +g 2 z +g 3 z 2 –O(1) to compute the max s(z) for each polynomial –O(n 4 ) in total S(z) z

36 Improved Variable Resolution Exact Algorithm Incremental computing –Computing polynomial coefficients O(n) for first smooth segment, O(1) for additional Introduce sorting cost –O(n log n) for each virtual corner –O(n 3 logn) total S(z) z

37 Improved Variable Resolution Exact Algorithm Diagonal Sweeping –No need to do sorting for each virtual corner –O(n) to get new sorted sequence –Total complexity O(n 3 ) x y O (a)(a) O y r1r1 r 2 x (b)(b) x (c)(c) y x (d)(d) O O r1r1 r1r1 Order of VCs y r1r1 r 2

38 Speed comparison Random inputs Brute force approach Using Incremental computing Using incremental computing and diagonal sweeping

39 Co-Opticon Problem Versions Fixed Resolution Exact Algorithm Variable Resolution Exact Algorithm Approximate Algorithm for Arbitrarily- Shaped Requests Distributed Algorithms D. Song, K. Goldberg, and A. Pashkevich, ShareCam Part II: Approximate and Distributed Algorithms for a Collaboratively Controlled Robotic Webcam, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2003.

40 Arbitrarily-Shaped Requested Frame Requested frames

41 Approximation Algorithm x y d Compute S(x,y) at lattice of sample points: w, h : width and height, g: size range

42 Approximation Bound Definition c * : Optimal frame : Optimal at lattice (Algorithm output)

43 Derive Approximation Bound c * : Optimal frame : Optimal at lattice (Algorithm output) : Smallest frame at lattice that encloses c *

44 Derive Approximation Bound c * : Optimal frame : Smallest frame at lattice that encloses c * fully enclose c*  What is the ratio between their objective functions if one candidate frame is enclosed by the other?

45 Approximation Bound Requested frames

46 Approximation Bound c Requested frames Candidate frame

47 Approximation Bound caca cbcb Requested frames Candidate frames

48 Approximation Bound caca cbcb Requested frames Candidate frames

49 Derive Approximation Bound c * : Optimal frame : Smallest frame at lattice that encloses c * What is the resolution ratio between a candidate frame and the smallest frame on the lattice that encloses it?

50 Approximation Algorithm caca cbcb d z : Lattice spacing in z axis d d

51 Approximation Algorithm –Run Time: –O(n /  3 ) c * : Optimal frame : Optimal at lattice (Algorithm output) : Smallest frame at lattice that encloses c *

52 Speed Comparison Random inputs Time (Sec.) #Requests

53 Enclosing the Optimal c * : Optimal frame : Frame at lattice that encloses c *

54 Cutting in Feasible Set x y z x y c’ Screen Space Solution Space Φ Φ c’

55 Branch and Bound on Lattice y z x kd z dzdz Layer 1 Layer 2 Layer 3 Survived nodes Deleted nodes

56 Speed Improvements n=5 n=20 n=40 ε

57 Improvement: Functional Box Sums Efficient reporting of [Zhang et al 2002]

58 Origin-Involved Functional Box Sums = __ +

59 Dominance Sums (14,8) (1,3)(9,3)

60 Dominance Sum Queries Data structure: –ECDF-tree Guttman (84) – ‘Simple’ updates when increasing zoom level

61 Co-Opticon Problem Versions Fixed Resolution Exact Algorithm Variable Resolution Exact Algorithm Approximate Algorithm for Arbitrarily- Shaped Requested Frame Distributed Algorithms

62 Distributed Algorithms

63 Distributed Algorithms Fixed Resolution Algorithms O(n 2 ) –Server O(nlogn) –Client O(n) Approximate Algorithm O(n/  3 ) –Server O(n+1/  3 ) –Client O(1/  3 ) –Robustness to dropouts…