UAV CINEMATOGRAPHY CONSTRAINTS IMPOSED BY VISUAL TARGET TRACKING Iason Karakostas, Ioannis Mademlis, Nikos Nikolaidis, Ioannis Pitas Dpt. Artificial Intelligence and Information Analysis Computer Science Aristotle University of Thessaloniki Greece

UAV in Cinematography UAVs have revolutionized aerial cinematography Can replace helicopters, cranes, etc. UAVs support autonomous functionalities based on machine learning and computer vision Autonomous UAVs may visually track and actively follow a specific target of interest We present common UAV target-tracking trajectories and shot types We study the constraints in maximum focal length for successful target tracking 16/10/2019

UAV in Cinematography We standardized and geometrically modeled a number of common, target-following UAV motion types We identified the compatible shot types We analytically determined the maximal permissible camera focal length, so that 2D visual tracking does not get lost, for each UAV motion type. 16/10/2019

UAV/Camera Shot Types The desired shot type is defined by the percentage of the video frame that is covered by the target Region Of Interest (ROI) Shot Type Percentage of ROI Extreme Long Shot (ELS) < 5 % Very Long Shot (VLS) 5 % – 20 % Long Shot (LS) 20 % – 40 % Medium Shot (MS) 40 % – 60 % Medium Close-Up (MCU) 60 % - 75 % Close-Up (CU) > 75 % 16/10/2019

UAV/Camera Motion Types 5 UAV industry- standard camera motion types are detailed and geometrically modeled Lateral Tracking Shot (LTS) Vertical Tracking Shot (VTS) Fly-Over (FLYOVER) Fly-By (FLYBY) Chase/Follow Shot (CHASE) 16/10/2019

UAV/Camera Motion Types LTS VTS FLYOVER FLYBY CHASE 16/10/2019

Constraints On Maximum Focal Length 2D visual tracking algorithms assume that the location of the target ROI center varies no more than a threshold 𝑅 𝑚𝑎𝑥 (in pixels) between successive video frames Focal length 𝑓 affects the permissible shot framing types For UAV cinematography it is important to determine the constraints on maximum focal length imposed by the needs of visual target tracking 16/10/2019

Constraints On Maximum Focal Length In a fully known 3D environment, if the target moves exactly as expected, its next 3D location can been predicted Thus, in this ideal scenario, central composition may always be retained by computing the appropriate LookAt vector at each time instance If target motion deviates from expected, its ROI may be displaced more than 𝑅 𝑚𝑎𝑥 on the next video frame 16/10/2019

Constraints On Maximum Focal Length We examine an entire shooting session as a sequence of repeated transitions between the “first” (𝑡 = 0) and the “second” video frame (𝑡 + 1 = 1) The target ROI center is meant to be fixed at the image center for all video frames, assuming accurate target velocity vector estimation at all times To study 𝑓 𝑚𝑎𝑥 we assume a maximum search radius 𝑅 𝑚𝑎𝑥 (in pixels) within which the ROI in 𝑡 + 1 must lie 16/10/2019

Constraints On Maximum Focal Length Maximum focal length is calculated based on the camera projection equations 𝑥 𝑑 𝑡+1 = 𝑜 𝑥 − 𝑓 𝑠 𝑥 𝑟 1 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑟 3 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑦 𝑑 𝑡+1 = 𝑜 𝑦 − 𝑓 𝑠 𝑦 𝑟 2 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑟 3 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑝 𝑡+1 , 𝑥 𝑡+1 : target/UAV expected position 𝑥 𝑑 , 𝑦 𝑑 : target center (pixel coordinates) 𝑜 𝑥 , 𝑜 𝑦 : image center (pixel coordinates) 𝑠 𝑥 , 𝑠 𝑦 : pixel size (mm) 𝑟 1 , 𝑟 2 , 𝑟 3 : rows of the rotation matrix 16/10/2019

Constraints On Maximum Focal Length Rotation Matrix the camera axis points directly at the target the unit vector of the k-axis for the Camera Coordinate System( 𝑟 3 ), can be obtained from 𝑥 𝑡+1 as follows: 𝑟 3 = − 𝑥 𝑡+1 𝑥 𝑡+1 𝑇 and 𝑟 1 ′ = 𝑘 ×− 𝑥 𝑡+1 𝑥 𝑡+1 𝑇 , 𝑟 2 ′ = − 𝑥 𝑡+1 𝑥 𝑡+1 × 𝑘 ×− 𝑥 𝑡+1 𝑥 𝑡+1 𝑇 𝑟 1 = 𝑟 1 ′ 𝑟 1 ′ , 𝑟 2 = 𝑟 1 ′ 𝑟 1 ′ 16/10/2019

Constraints On Maximum Focal Length Using the limit constraint 𝑅 𝑡+1 = 𝑅 𝑚𝑎𝑥 𝑅 𝑚𝑎𝑥 = 𝑥 𝑑 𝑡+1 − 𝑜 𝑥 2 + 𝑦 𝑑 𝑡+1 − 𝑜 𝑦 2 Using projection equations, 𝑓 𝑚𝑎𝑥 is given by: 𝑓 𝑚𝑎𝑥 = 𝑅 𝑚𝑎𝑥 𝑑 𝑡 ′ 𝑠 𝑥 𝑠 𝑦 𝐸 1 +𝐹 𝑥 𝑡 ′ 2 𝑠 𝑥 𝑞 𝑡3 𝑑 𝑡 ′ 2 − 𝑠 𝑥 𝑥 𝑡 ′ 3 𝐸 2 2 + 𝑠 𝑦 2 𝐸 3 2 𝑥 𝑡 ′ 2 where 𝐸 1 =− 𝑞 𝑡1 𝑥 𝑡 ′ 1 − 𝑞 𝑡2 𝑥 𝑡 ′ 2 − 𝑞 𝑡3 𝑥 𝑡 ′ 3 , 𝐸 2 = 𝑞 𝑡1 𝑥 𝑡 ′ 1 + 𝑞 𝑡2 𝑥 𝑡 ′ 2 , 𝐸 3 = 𝑞 𝑡2 𝑥 𝑡 ′ 1 − 𝑞 𝑡1 𝑥 𝑡 ′ 2 . 16/10/2019

Constraints On Maximum Focal Length Experimental cases 8 cases for the deviation vector 𝑞 𝑡 𝑞 𝑡1 =[5, 0, 𝑞 𝑡3 ] 𝑞 𝑡2 =[−5, 0, 𝑞 𝑡3 ] 𝑞 𝑡3 =[0, 5, 𝑞 𝑡3 ] 𝑞 𝑡4 =[0, −5, 𝑞 𝑡3 ] 𝑞 𝑡5 =[5, 5, 𝑞 𝑡3 ] 𝑞 𝑡6 =[−5, −5, 𝑞 𝑡3 ] 𝑞 𝑡7 =[−5, 5, 𝑞 𝑡3 ] 𝑞 𝑡8 =[5, −5, 𝑞 𝑡3 ] 16/10/2019

Constraints On Maximum Focal Length Lateral Tracking Shot UAV position is given by 𝑥 𝑡+1 = 0, 𝑥 𝑡2 ,0 𝑇 Target position is given by 𝑝 𝑡+1 = 𝑞 𝑡1 𝐹 , 𝑞 𝑡2 𝐹 , 𝑞 𝑡3 𝐹 𝑇 , where 𝐹 is the camera framerate Maximum focal length for the LTS is now given by 𝑓 𝑚𝑎𝑥 = 𝑅 𝑚𝑎𝑥 𝑠 𝑥 𝑠 𝑦 𝑞 𝑡2 −𝐹 𝑥 𝑡2 𝑠 𝑦 2 𝑞 𝑡1 2 + 𝑠 𝑥 2 𝑞 𝑡3 2 16/10/2019

Constraints On Maximum Focal Length Variations in target altitude affect all study cases 1 – 8 when 𝑞 𝑡3 =0 the projected ROI center will not change in pixel coordinates, if the target approaches or goes away from the UAV. Due to the position of the UAV, target acceleration and deceleration have identical impact on 𝑓 𝑚𝑎𝑥 Variation of 𝑓𝑚𝑎𝑥 against 𝑞𝑡3 for LTS 16/10/2019

Constraints On Maximum Focal Length Cases 1-2: UAV approaches the target and the maximum focal length decreases, before increasing again as the UAV is flying parallel to the 𝒊-axis. When the drone is positioned far from the target, any change in target velocity corresponds to a small change in the distance between the UAV and the target. Cases 3-4: target deviates from its expected position but remains on the 𝒋-axis 𝑓 𝑚𝑎𝑥 increases with distance between the UAV and the target. 𝑓 𝑚𝑎𝑥 slightly increases when the UAV is very close to the target. Cases 5-8: 𝑓 𝑚𝑎𝑥 depends on the angle between the LookAt vector and the 𝒊-axis: it has lower values when this angle is close to 𝜋 2 (𝑡=10). Variation of 𝑓𝑚𝑎𝑥 against 𝑡 for FLYBY 16/10/2019

Constraints On Maximum Focal Length FLYOVER CHASE VTS 16/10/2019

Conclusions Industry-standard target-tracking UAV/camera motion types have been formalized and geometrically modelled Maximum focal length constraints for computer vision-assisted UAV physical target following have been extracted The derived formulas can be readily employed as low-level rules in intelligent UAV shooting and cinematography planning systems 16/10/2019

Acknowledgement The research leading to these results has received funding from the European Union’s European Union Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE). 16/10/2019

Thank you! Q&A