1. UAV CINEMATOGRAPHY CONSTRAINTS IMPOSED BY VISUAL TARGET TRACKING
Iason Karakostas, Ioannis Mademlis, Nikos Nikolaidis, Ioannis Pitas
Dpt. Artificial Intelligence and Information Analysis
Computer Science
Aristotle University of Thessaloniki
Greece

2. UAV in Cinematography UAVs have revolutionized aerial cinematography
Can replace helicopters, cranes, etc.
UAVs support autonomous functionalities based on machine learning and computer vision
Autonomous UAVs may visually track and actively follow a specific target of interest
We present common UAV target-tracking trajectories and shot types
We study the constraints in maximum focal length for successful target tracking
16/10/2019

3. UAV in Cinematography
We standardized and geometrically modeled a number of common, target-following UAV motion types
We identified the compatible shot types
We analytically determined the maximal permissible camera focal length, so that 2D visual tracking does not get lost, for each UAV motion type.
16/10/2019

4. UAV/Camera Shot Types
The desired shot type is defined by the percentage of the video frame that is covered by the target Region Of Interest (ROI)
Shot Type
Percentage of ROI
Extreme Long Shot (ELS)
< 5 %
Very Long Shot (VLS)
5 % – 20 %
Long Shot (LS)
20 % – 40 %
Medium Shot (MS)
40 % – 60 %
Medium Close-Up (MCU)
60 % - 75 %
Close-Up (CU)
> 75 %
16/10/2019

5. UAV/Camera Motion Types
5 UAV industry- standard camera motion types are detailed and geometrically modeled
Lateral Tracking Shot (LTS)
Vertical Tracking Shot (VTS)
Fly-Over (FLYOVER)
Fly-By (FLYBY)
Chase/Follow Shot (CHASE)
16/10/2019

6. UAV/Camera Motion Types
LTS
VTS
FLYOVER
FLYBY
CHASE
16/10/2019

7. Constraints On Maximum Focal Length
2D visual tracking algorithms assume that the location of the target ROI center varies no more than a threshold 𝑅 𝑚𝑎𝑥 (in pixels) between successive video frames
Focal length 𝑓 affects the permissible shot framing types
For UAV cinematography it is important to determine the constraints on maximum focal length imposed by the needs of visual target tracking
16/10/2019

8. Constraints On Maximum Focal Length
In a fully known 3D environment, if the target moves exactly as expected, its next 3D location can been predicted
Thus, in this ideal scenario, central composition may always be retained by computing the appropriate LookAt vector at each time instance
If target motion deviates from expected, its ROI may be displaced more than 𝑅 𝑚𝑎𝑥 on the next video frame
16/10/2019

9. Constraints On Maximum Focal Length
We examine an entire shooting session as a sequence of repeated transitions between the “first” (𝑡 = 0) and the “second” video frame (𝑡 + 1 = 1)
The target ROI center is meant to be fixed at the image center for all video frames, assuming accurate target velocity vector estimation at all times
To study 𝑓 𝑚𝑎𝑥 we assume a maximum search radius 𝑅 𝑚𝑎𝑥 (in pixels) within which the ROI in 𝑡 + 1 must lie
16/10/2019

10. Constraints On Maximum Focal Length
Maximum focal length is calculated based on the camera projection equations 𝑥 𝑑 𝑡+1 = 𝑜 𝑥 − 𝑓 𝑠 𝑥 𝑟 1 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑟 3 𝑡 𝑝 𝑡+1 − 𝑥 𝑡 𝑦 𝑑 𝑡+1 = 𝑜 𝑦 − 𝑓 𝑠 𝑦 𝑟 2 𝑡 𝑝 𝑡+1 − 𝑥 𝑡+1 𝑟 3 𝑡 𝑝 𝑡+1 − 𝑥 𝑡 𝑝 𝑡+1 , 𝑥 𝑡+1 : target/UAV expected position 𝑥 𝑑 , 𝑦 𝑑 : target center (pixel coordinates) 𝑜 𝑥 , 𝑜 𝑦 : image center (pixel coordinates) 𝑠 𝑥 , 𝑠 𝑦 : pixel size (mm) 𝑟 1 , 𝑟 2 , 𝑟 3 : rows of the rotation matrix
16/10/2019

11. Constraints On Maximum Focal Length
Rotation Matrix
the camera axis points directly at the target the unit vector of the k-axis for the Camera Coordinate System( 𝑟 3 ), can be obtained from 𝑥 𝑡+1 as follows: 𝑟 3 = − 𝑥 𝑡+1 𝑥 𝑡 𝑇 and 𝑟 1 ′ = 𝑘 ×− 𝑥 𝑡+1 𝑥 𝑡 𝑇 , 𝑟 2 ′ = − 𝑥 𝑡+1 𝑥 𝑡+1 × 𝑘 ×− 𝑥 𝑡+1 𝑥 𝑡 𝑇 𝑟 1 = 𝑟 1 ′ 𝑟 1 ′ , 𝑟 2 = 𝑟 1 ′ 𝑟 1 ′
16/10/2019

12. Constraints On Maximum Focal Length
Using the limit constraint 𝑅 𝑡+1 = 𝑅 𝑚𝑎𝑥 𝑅 𝑚𝑎𝑥 = 𝑥 𝑑 𝑡+1 − 𝑜 𝑥 𝑦 𝑑 𝑡+1 − 𝑜 𝑦 2
Using projection equations, 𝑓 𝑚𝑎𝑥 is given by: 𝑓 𝑚𝑎𝑥 = 𝑅 𝑚𝑎𝑥 𝑑 𝑡 ′ 𝑠 𝑥 𝑠 𝑦 𝐸 1 +𝐹 𝑥 𝑡 ′ 𝑠 𝑥 𝑞 𝑡3 𝑑 𝑡 ′ 2 − 𝑠 𝑥 𝑥 𝑡 ′ 3 𝐸 𝑠 𝑦 2 𝐸 𝑥 𝑡 ′ where 𝐸 1 =− 𝑞 𝑡1 𝑥 𝑡 ′ 1 − 𝑞 𝑡2 𝑥 𝑡 ′ 2 − 𝑞 𝑡3 𝑥 𝑡 ′ 3 , 𝐸 2 = 𝑞 𝑡1 𝑥 𝑡 ′ 1 + 𝑞 𝑡2 𝑥 𝑡 ′ 2 , 𝐸 3 = 𝑞 𝑡2 𝑥 𝑡 ′ 1 − 𝑞 𝑡1 𝑥 𝑡 ′ 2 .
16/10/2019

13. Constraints On Maximum Focal Length
Experimental cases
8 cases for the deviation vector 𝑞 𝑡
𝑞 𝑡1 =[5, 0, 𝑞 𝑡3 ]
𝑞 𝑡2 =[−5, 0, 𝑞 𝑡3 ]
𝑞 𝑡3 =[0, 5, 𝑞 𝑡3 ]
𝑞 𝑡4 =[0, −5, 𝑞 𝑡3 ]
𝑞 𝑡5 =[5, 5, 𝑞 𝑡3 ]
𝑞 𝑡6 =[−5, −5, 𝑞 𝑡3 ]
𝑞 𝑡7 =[−5, 5, 𝑞 𝑡3 ]
𝑞 𝑡8 =[5, −5, 𝑞 𝑡3 ]
16/10/2019

14. Constraints On Maximum Focal Length
Lateral Tracking Shot
UAV position is given by 𝑥 𝑡+1 = 0, 𝑥 𝑡2 ,0 𝑇
Target position is given by 𝑝 𝑡+1 = 𝑞 𝑡1 𝐹 , 𝑞 𝑡2 𝐹 , 𝑞 𝑡3 𝐹 𝑇 , where 𝐹 is the camera framerate
Maximum focal length for the LTS is now given by 𝑓 𝑚𝑎𝑥 = 𝑅 𝑚𝑎𝑥 𝑠 𝑥 𝑠 𝑦 𝑞 𝑡2 −𝐹 𝑥 𝑡 𝑠 𝑦 2 𝑞 𝑡1 2 + 𝑠 𝑥 2 𝑞 𝑡3 2
16/10/2019

15. Constraints On Maximum Focal Length
Variations in target altitude affect all study cases 1 – 8
when 𝑞 𝑡3 =0 the projected ROI center will not change in pixel coordinates, if the target approaches or goes away from the UAV.
Due to the position of the UAV, target acceleration and deceleration have identical impact on 𝑓 𝑚𝑎𝑥
Variation of 𝑓𝑚𝑎𝑥 against 𝑞𝑡3 for LTS
16/10/2019

16. Constraints On Maximum Focal Length
Cases 1-2: UAV approaches the target and the maximum focal length decreases, before increasing again as the UAV is flying parallel to the 𝒊-axis. When the drone is positioned far from the target, any change in target velocity corresponds to a small change in the distance between the UAV and the target.
Cases 3-4:
target deviates from its expected position but remains on the 𝒋-axis
𝑓 𝑚𝑎𝑥 increases with distance between the UAV and the target.
𝑓 𝑚𝑎𝑥 slightly increases when the UAV is very close to the target.
Cases 5-8:
𝑓 𝑚𝑎𝑥 depends on the angle between the LookAt vector and the 𝒊-axis: it has lower values when this angle is close to 𝜋 2 (𝑡=10).
Variation of 𝑓𝑚𝑎𝑥 against 𝑡 for FLYBY
16/10/2019

17. Constraints On Maximum Focal Length
FLYOVER
CHASE
VTS
16/10/2019

18. Conclusions
Industry-standard target-tracking UAV/camera motion types have been formalized and geometrically modelled
Maximum focal length constraints for computer vision-assisted UAV physical target following have been extracted
The derived formulas can be readily employed as low-level rules in intelligent UAV shooting and cinematography planning systems
16/10/2019

19. Acknowledgement
The research leading to these results has received funding from the European Union’s European Union Horizon 2020 research and innovation programme under grant agreement No (MULTIDRONE).
16/10/2019

20. Thank you!
Q&A