1. Rotational Rectification Network (R2N):
Enabling Pedestrian Detection for Mobile Vision
Xinshuo Weng1, Shangxuan Wu1, Fares Beainy2, Kris M. Kitani1
1Carnegie Mellon University, 2Volvo Construction Equipment
WACV 2018, Lake Tahoe
Hi everyone, I’m Xinshuo Weng from CMU. This work called R2N, rotational rectification network, is done with Shangxuan Wu, Kris Kitani from CMU and Fares Beainy from Volvo.

2. Pedestrian Detection
Pedestrian detection is a very hot topic in computer vision over the past decade. Given an input image, the goal is to localize the pedestrians.

3. Pedestrian Detection Results on Caltech dataset
Lots of works have been done in this area. This figure is a summary of results on Caltech dataset by You can see that the best entry at that time can already achieve an average miss rate less than 10%, which is pretty good.
Zhang et al. Is Faster R-CNN Doing Well for Pedestrian Detection? ECCV, 2016.

4. Arbitrary-Oriented Pedestrian Detection
Okay, now. What if we rotate the input image and run the same detector? What do you think will happen?

5. Arbitrary-Oriented Pedestrian Detection
As you can see in the bottom row, even if the detector is exactly the same as before and we just add a little bit angle to the input image, it leads to inferior detection performance. The key difference here is that the pedestrians are not upright any more after the rotation.

6. Arbitrary-Oriented Pedestrian Detection
Random failure cases on Caltech dataset.
Here are some more random failure cases. When there are no bounding box, it means that the detector totally fails.

7. Why is it interesting? Imagine the cases: Mobile phones
So, why is it interesting? Why do we need to study the arbitrary-oriented pedestrian detection? Well, the truth is, in the real world there are lots of cases where the pedestrians are not upright.
This is an image I took yesterday during the poster session. Obviously, people are not upright even if I did it intentionally. But what I am trying to say is that there is a big chance to have an angled picture when using mobile phones.

8. Why is it interesting? Imagine the cases: Mobile phones UAVs/drones
Similar for cameras on the drones.

9. Why is it interesting? Imagine the cases: Mobile phones UAVs/drones
Construction vehicles on a rugged terrain
Same things happen when driving a construction vehicle on a rugged road.

10. Why is it interesting? Imagine the cases: Mobile phones UAVs/drones
Construction vehicles on a rugged terrain
Wearable cameras ….
And of course, the wearable cameras, such as gopro.

11. Why is it interesting? Imagine the cases: Mobile phones UAVs/drones
Construction vehicles on a rugged terrain
Wearable cameras ….
Camera orientation can be very flexible with respect to the ground in the real world.
In all of these cases in the real world, the camera orientation can be flexible to the ground.

12. Modeling Rotation Invariance or Equivariance
In order to have the an expected results shown in the figure, it is straightforward to think about modeling the rotation invariance or equivarience in the network.

13. Modelling Rotation Invariance/Equivariance
Rotating the inputs
Data augmentation
TI-Pooling [Laptev et al CVPR’ 16] ….
Cons:
Low efficiency
More parameters
Rotating the filters
Changing sampling grids
Most methods can be categorized into 3 families, rotating the inputs, filters or changing the sampling grids. Obviously, data augmentation belongs to the first category. The main disadvantage of the first family is that they often suffer from learning lots of redundant filters with different angles in low-level layers.

14. Modelling Rotation Invariance/Equivariance
Rotating the inputs
Data augmentation
TI-Pooling [Laptev et al, CVPR’ 16] ….
Cons:
Low efficiency
More parameters
Rotating the filters
RotEqNet [Marcos et al, ICCV’ 17]
ORNs [Zhou et al, CVPR’ 17] ….
Cons:
Approximated rotations
Memory issues
Changing sampling grids
We can instead rotate the filters, which however produces more intermediate activation maps with different angles. So we can only have a few filters due to memory issues. Usually, people use 4 filters with an interval of 90 degrees. So the small rotation can only be approximated.

15. Modelling Rotation Invariance/Equivariance
Rotating the inputs
Data augmentation
TI-Pooling [Laptev et al, CVPR’ 16] ….
Cons:
Low efficiency
More parameters
Rotating the filters
RotEqNet [Marcos et al, ICCV’ 17]
ORNs [Zhou et al, CVPR’ 17] ….
Cons:
Approximated rotations
Memory issues
Changing sampling grids
Spatial Transformer [Jaderberg et al, NIPS’ 15]
Deformable ConvNets [Dai et al, ICCV’ 17]
GPPooling (Ours) ….
The third family is to change the sampling grid. In this way, rotation can be accurately modeled. This is usually a smarter way. Our proposed method called, Global polar pooling, or GPPooling, also belongs to the third family.

16. Global Polar Pooling (GPPooling)
This is a demo to show what activations can be produced on the right by our GPPooling when rotating the inputs on the left. The key observation here is that the rotational changes in the inputs leads to translational shifts in the output activations, which can be easily recognized by higher-level layers because, as we all know, the translation equivariance is naturally encoded in CNNs.
Inputs
Activations

17. GPPooling vs Pooling GPPooling Pooling
The core idea of GPPooling is very simple. We basically define the sampling grid in a polar coordinate instead of a rectangular one compared to the general pooling layer. The stride and kernel size are defined in a similar spirit, but along radial and angular axes. If you take the max from the sampled grid shown in the figure, so you got max-GPPooling layer. The kernel size and stride can be defined to a very small value if one wants to model a small amount of rotation.
Noh et al. Learning Deconvolution Network for Semantic Segmentation? ICCV, 2015.

18. What is Rotational Rectification Network (R2N)?
R2N = Rotation Estimation Module (including GPPooling) + Spatial Transformer
In order to solve the oriented pedestrian detection, we propose the R2N, which is the light blue box shown in the figure. R2N is basically composed of a rotation estimation module (the dark blue one) and a spatial transformer (the green one). The detector in the yellow box can be any general detection framework, such as faster-rcnn. If you look at the rotation estimation module closely, we use the proposed GPPooling inside it to capture any rotation present in the inputs. The estimated rotation is then passed through transformer to rectify the image features such that the pedestrians are converted to be upright in the feature space. In this way, you can detect the angled pedestrians as easily as detect the upright pedestrians.

19. Results
Here are some example results. You can get a much better tight bounding box with an estimated angle here.

20. Take Home Messages
GPPooling can be used to model global rotation equivariance/invariance in general CNNs.
R2N is easy to plug in and improves the performance on oriented detection without bells and whistles.
Some take home messages. If you want to model the rotation equivariance, use GPPooling. It is just a counterpart of general pooling layer. If you have a task of detecting oriented objects, have a try with R2N. If you think our work is interesting, please remember or mark down our title, rotational rectification network and stop by our poster. Thanks!