1. Optical Flow and Obstacle Avoidance
Computer Vision Research Group

2. Optical Flow Concepts
Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera. It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second.
The image shows a ball moving in 5 consecutive frames. The arrow shows its displacement vector.

3. Optical Flow Problem

4. Optical Flow Computation
Optical flow works on several assumptions:
The pixel intensities of an object do not change between consecutive frames.
Neighbouring pixels have similar motion.
Consider a pixel in first frame I(x,y,t). It moves by distance (dx,dy) in next frame taken after dt time. So since those pixels are the same and intensity does not change, we can say
I(x,y,t) = I(x+dx, y+dy, t+dt)
Then take taylor series approximation of right-hand side, remove common terms and divide by dt
Where:

5. Optical Flow Computation

6. Optical Flow Computation
Previous equation is called Optical Flow equation. In it, we can find fx ­and fy , they are image gradients. Similarly, ft is the gradient along time. But (u,v) are unknown. We cannot solve this one equation with two unknown variables.

7. Sparse Optical Flow Computation
By Lucas-Kanade:
We have seen an assumption before, that all the neighbouring pixels will have similar motion. Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion. We can find fx ­, fy and ft for these 9 points. So now our problem becomes solving 9 equations with two unknown variables which is over-determined. A better solution is obtained with least square fit method. Below is the final solution which is two equation-two unknown problem and solve to get the solution.

8. Demo
We take the first frame, detect some Shi-Tomasi corner points in it, then we iteratively track those points using Lucas-Kanade optical flow
Demo

9. Dense Optical Flow Computation
By Gunner Farneback:
It computes the optical flow for all the points in the frame.
Demo

10. Problem

11. Challanges

12. Usage of Optical flow Ground Detection
Goal:
Segment image into 2 layers, ground plane and other. Ground plane area represent traversable path, the other layer represents potential obstacles.

13. Ground Detection A Suggested Method
the idea is based on computing a theoretical optical flow model for ground pixels, and compare each pixel in input frames with the model to label it as ground pixel or other object pixel.

14. Ground Optical Flow Model
assuming a pinhole camera model, and a robot with known linear and rotational velocity, the optical flow of ground pixels is calculated from the projection equation of ground points from world frame to image frame.

15. Ground Optical Flow Model

16. Ground Optical Flow Model
camera projection matrix:
projection equation of ground points from world to image coordinates

17. Ground Optical Flow Model
representation as homography (plane to plane projection)
differentiation of previous equation

18. Ground Optical Flow Model
simplification:
from eqn. 2 and 4:
optical flow vector:

19. Ground Optical Flow Model
the homography matrix:

20. Ground Motion Segmentation
for each pixel in an input frame, we can calculate theoretical optical flow, if its actual optical flow is similar to the theoretical one, then it’s labeled as ground pixel.

21. Ground Motion Segmentation Actual Optical Flow Issues
issues in computing actual optical flow for each pixel as claimed by the paper in 2006
1- time complexity
2- accuracy
3- comparison function between actual and theoretical optical flow

22. Ground Motion Segmentation A Generative Method
for each pixel , calculate theoretical optical flow, and and determine if it’s a possible displacement according to the model, by computing similarity between pixel (u,v) in image I(t) , and pixel (u,v) t * opticalFlowVector(u,v) in image
I(t- t). using SSD and SAD similarity measures

23. Results

24. Usage of Optical flow Motion Segmentation
Goal:
Segment the scene into coherent regions based on the real-world motion of the objects in it.

25. Usage of Optical flow Motion Segmentation

26. Motion Segmentation Scenarios
Stationary Camera
→ Background Subtraction: Gaussian, Bayesian, Non-parametric Models.
→ Camera motion causes most image pixels to move, pixelwise models are no longer adequate.

27. Motion Segmentation Scenarios
Moving Camera
→ Optical flow
Direct:Clustering
Indirect: Camera motion compensation

28. Major Drawback
Objects at different depths will appear to have different labels since closer objects have a higher optical flow.
Post-processing is required to merge smaller segments into one background

30. University of Massachusetts, Amherst
Coherent Motion Segmentation in Moving Camera Videos using Optical Flow Orientations
Manjunath Narayana Allen Hanson Erik Learned-Miller
University of Massachusetts, Amherst

31. Coherent Motion Segmentation in Moving Camera Videos using Optical Flow Orientations
Library of predicted orientation fields (46) which cover a large space of possible translations.
Maps each pixel to one of the orientation fields.
Pixels with optical flow magnitude less than a threshold are disregarded.
Use a probabilistic model (Dirichlet model) for clustering.

32. Improvements
Color appearance model (Gaussian) for the background and foreground at each pixel. (Motion compensated)
A uniform color distribution is mixed into the color likelihood.
Posterior computation to improve confidence from previous frames. (Motion compensated)

33. Results Hopkins data set (26 videos) SegTrack segmentation data set.
Complex backgrounds

34. Results Among the SegTrack data set, three
videos (marked with *) have multiple moving objects, but the ground truth intended for tracking analysis marks only one primary object as the foreground
Complex backgrounds contained camera rotations.

35. Videos Hopkins: Marple1 SegTrack: Birdfall2 Penguin*
Complex Background: Parking

36. Disadvantages Object motion matches the camera motion
Assumption of translational motion camera (tx,ty,tz)

37. Obstacle Avoidance “Balance Strategy”
Motion parallax Concept.
Car and plane example.
How to avoid obstacles.
Experiments and conditions.
Criterias for each algorithm.
Different algorithms with demos.

38. Motion Parallax
Motion parallax is a concept where due to the movement of the observer or the camera in our case, the position of pixels move from a frame to the other either in x or y direction.
The closer pixels will produce higher optical flow than the farther pixels which gives us a sense of depth.

40. Example
(car)
Try to drive a car in a high speed and look at your surrounding.You will notice 2 things :
The trees along the road which are close to your car are passing by very fast.
The trees that are far away pass by slowly.

41. Example continued (car)
Conclusion:
Pixels of the closer objects move with a speed (dx/dt , dy/dt) bigger than farther objects.
We can deduce an avoiding strategy based on that observation.

42. Example Continued (Plane)
Try to look at the sky and search for a plane.
Once you find it you will notice :
From the ground the plane is very slow.
The plane is almost like standing still and not moving.

43. Example continued (plane)
Conclusion :
Same conclusion as the car example.
Since the plane is at very high height the dx/dt and dy/dt of each pixel is very small.

44. How to avoid obstacles?
We can avoid hitting an obstacle by simply moving toward the part of the frame with the least optical flow value.
Small optical flow value means a far object.

45. How to avoid obstacles? continued
Steps :
For each 2 consecutive frames from the video captured repeat the following :
Compute the optical flow between the 2 frames in x and y direction (dx/dt,dy/dt).
Assuming a dense algorithm each pixel will have an optical flow value. so divide them into

46. How to avoid obstacles? continued
two halves Left and right.
3. Add the values of the right half together y and x direction.
4. Add the values of the left half together y and x direction.
5. Check which half has a smaller value and move toward it.

47. Experiments
For each experiment some conditions were applied or tested to test the accuracy of the algorithm :
When something like a left hand for example is moving will the algorithm compute the correct flow. Will the left half value of the optical flow have a bigger value than the right.

48. Experiments continued
2. Experiment when there is no source of light at all.The goal is that the flow predicted from this is that the left and the right flow will be almost the same.
3. Experiment when there is a source of light but no movement. The goal is to check if the light has an effect when computing optical flow.

49. Experiments Continued
The predicted flow of the right half should be near to the right half however if that is not the case that means the light has an effect on computing optical flow. Thats the difference between this experiment and the previous one “Light Effect”

50. Experiments continued
4. After testing all the conditions above a demo is provided to show the algorithm in real life.
5. Analysis for the algorithm is provided using the following criteria :

51. Criteria Accuracy on kitti benchmarks.
Running Time between 2 frames without optimizations (C++ if code provided)
Running Time between 2 frames with optimizations (C++ if code provided)
Running time between 2 frames (Matlab if code provided)

52. Criteria continued Dense algorithm (Yes /No) Resolution of the frame.
A Final table will hold all the algorithm and there analysis like so :
Algorithm Name :
Page Reference :
Accuracy on KITTI :
Running time C++ without optimizations :
Running time C++ with optimizations :
Running time matlab :
Dense algorithm
Yes/No :
Resolution of each frame :

53. Algorithms I Algorithm Name : C. Liu. Beyond Pixels
Reference : C. Liu. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology. May

54. Algorithm I continued Dense Algorithm. Resolution : 240x320.
Running time between two frames on average : 5.1 sec
Accuracy on KITTI :

56. Algorithm I Continued Experiment 1 :
My right hand was moving quickly but my left was standing still. Note: The algorithm flips the image so my right hand will be in the left part of the image. So the expected flow values will be that the left sum is bigger.

57. Algorithm I
continued
Frame 1
Frame 2

58. Algorithm I continued LeftSum : 2.8536e+04 RightSum : 1.7906e+04
Output: Turn Right
which is the expected result.

59. Algorithm I continued Experiment 2 :
When there is no source of light. The goal is to check that the algorithm is working. The predicted result should be that the left and right flow are almost the same.

60. Algorithm I
continued
Frame 1
Frame 2

61. Algorithm I continued LeftSum : 34.3381 RightSum : 37.617
As you can see the values are extremly close which is what we want.

62. Algorithm I continued Experiment 3 :
Source of light but no movements. The goal is to check if the light has an effect on computing the optical flow. The predicted result is that the left and right values are almost the same.

63. Algorithm I
continued
Frame 1
Frame 2

64. Algorithm I continued LeftSum : 2.6426e+04 RightSum : 1.1546e+05
As we can notice the difference is big which means that the light effect has an effect on values don’t know why so I redid the same experiment to observe the values if each time they are the same or not. Here are the values

65. Algorithm I continued for 2 more times. LeftSum1 : 1.2517e+03
RightSum1 : e+03
They are pretty close.
LeftSum2 : e+03
RightSum2 : e+04
They are not close.The behaviour undefined.

66. Algorithm I continued Video Demo :
To make the video easier when the algorithm says to turn left this sound will play:
when turn right this sound will play:

68. Turn Left

70. Turn Right

72. Turn Left

74. Turn Right

76. Turn Left

78. Turn Right

80. Turn Right