1. An Edge detection and HPF-based Intelligent Space – A Network based Integrated Navigation System By, Rachana Ashok Gupta Under the direction of Dr. Mo-Yuen Chow

2. NC STATE UNIVERSITY ADAC, NC State University2 Overview of presentation l What are network based integrated navigation system (NBINS) in brief followed by Abstract for this research. l Introduction to iSpace – iSpace components and modules – Limitations and Scope of improvement as a NBINS l The new structure of iSpace – New modules (edge detection and HPF planner) – Advantages and improvements achieved over the old structure l Results and Discussions

3. NC STATE UNIVERSITY ADAC, NC State University3 Network based integrated navigation system l Advantages – Remotely control over a long-distance. – Efficient to fuse global information. – Scalability - Easy to add more sensors and UGVs with very little cost and without heavy structural changes. l Applications – Manufacturing plant monitoring – Nursing homes or hospitals – Tele-robotics & Tele-operation etc l Issues to be considered – Network and processing delay – Data sharing and – Interfacing Network based Integrated Navigation systems – Different modules combined together to guide a UGV (Unmanned Ground Vehicle) from one point to another in the space of interest (  2 ), where the navigational intelligence lies on a main controller away from the UGV.

4. NC STATE UNIVERSITY ADAC, NC State University4 What is Intelligent Space l A new concept to effectively use distributed sensors, actuators, robots, computing processors, and information technology over a physically and/or virtually connected space. For examples, a room, a corridor, a hospital, an office, or a planet. l It fuses global information within the space of interest to make intelligent operation decision such as how to move a mobile robot effectively from one location to another. Human-machine interaction in iSpace

5. NC STATE UNIVERSITY ADAC, NC State University5 iSpace as a NBINS l Components – Overhead network camera – Network controller with graphical user interface – A Differential drive UGV as the navigator – Computer Network (IP) l On the main network controller – Image acquisition – Image processing – Path generation – Path tracking controller l Graphic User Interface (GUI) – Any remote Computing interface in the world (Internet)

6. NC STATE UNIVERSITY ADAC, NC State University6 Template Matching l Image acquisition – Top view l Image processing – Image Thresholding – Black and white – Template matching – Circles used for rotation invariance – Draws the circle around obstacle with safety margin radius of r safe. f B – Template image

7. NC STATE UNIVERSITY ADAC, NC State University7 Path generation l Find a path from the starting point A to the end point B for the UGV – The path of the UGV should be as short as possible (minimize time) – The path of the UGV should not collide with any obstructions l Fast Marching Method (by J.A. Sethian, Dept. of Mathematics, UC Berkeley) – A numerical technique that counts the shortest distance from a point to the original point with a shortest distance update algorithm

8. NC STATE UNIVERSITY ADAC, NC State University8 Path tracking The path tracking algorithm runs in every control loop and adjusts the speed and turn rate of the UGV to track the generated path. 1. Calculate the closest point on the reference path from the current UGV position (xc, yc,  c). 2. Pick a reference point (xref, yref) on the generated path that is a set distance (d 0 ) 3. speed and turn rate for the UGV to reach the reference point given its current position and orientation.

9. NC STATE UNIVERSITY ADAC, NC State University9 Time delay issue l Network delay component – Sensor to controller delay (Image Acquisition) – Controller to actuator delay (Commands from controller to the UGV) l Processing or computational delay component – Non-real time Computational delay (Initial image processing, path planning) – Real time Computational delay (continuous image processing, motion control)

10. NC STATE UNIVERSITY ADAC, NC State University10 Limitations - Template matching Insufficient safety margin Conservative safety margin l Template matching output – Location co-ordinates of » UGV » Obstacles l Limitations – Obstacles are from a priori set – No knowledge about shape and size – Either Insufficient or conservative r safe – Leading to inefficient or non-optimal path planning All these make the system restricted to operate in only a few environmental patterns. r safe

11. NC STATE UNIVERSITY ADAC, NC State University11 Limitations with Fast Marching l Implements and maintains a binary tree through out the path generation. l O(N(LogN)) problem and memory intensive. The time and the number of loop iterations are dependent on the respective position of the destination from the source. l Operations involved are – search, distance calculations (squares and sqaure root functions) l Every iteration – need of check whether the TRIAL point is outside the r safe margin of recognized obstacle.

12. NC STATE UNIVERSITY ADAC, NC State University12 Limitations with path planning l Navigation problem is looked upon as a path tracking problem and therefore the reference path generation is mandatory. l The reference point generation on the path for the UGV (off the path) does not consider the obstacle avoidance. l Quadratic curve controller needs current position and the reference point to calculate speed. Complete reference path is not needed. l Real time computation to find the closest point on the reference path Point to point guidance function

13. NC STATE UNIVERSITY ADAC, NC State University13 Important points Delay tolerance, efficiency, accuracy, generality and optimality of iSpace depend upon the following factors. l Fast, efficient and generic enough Image processing algorithm with different category of obstacle maps for the navigation system. l The path planning algorithm will decide – Optimality and length of the path generated l Path tracking algorithm has to continuously consider the obstacle avoidance for a navigation system – Probability to hit an obstacle. – The time required to track that path l Compatibility and data flow between different modules is also a key factor for an integrated system.

14. NC STATE UNIVERSITY ADAC, NC State University14 New Structure for iSpace Points of emphasis – 1. Processing and computation delay, increase in the efficiency, generality, optimality with each added new module to create a suitable platform for NBINS. 2. Creating a homogenous structure by putting edge detection, HPF planner and quadratic curve fitting path tracking controller - three heterogeneous systems together for the first time to create a network based system is the novel contribution to the network based integrated navigation system area.

15. NC STATE UNIVERSITY ADAC, NC State University15 Edge Detection Image I Laplacian of Gaussian  2 G Gradient of Gaussian  G Zero Crossing Magnitude Threshold (C) Edge map E E(x i, y j ) = 1 if (x i, y j )   B = 0 if (x i, y j )   B for all (i, j) Where E(x, y) is the image representing the edge map and  B is the set of edge points including the boundary points for all obstacles in workspace.

16. NC STATE UNIVERSITY ADAC, NC State University16 2-D Harmonic Potential Field –  2 is the Laplace operator,  is the workspace of the UGV (    2 ),  is the boundary of the obstacles (output of the edge detection stage), and is (x T, y T ) the target point. The obstacles were represented by the repelling force and the point of destination was represented by the attractive force. – The potential function in closed contour of  will converge to a constant potential – The obstacle free path to the target is generated by traversing the negative gradient(  ) i.e. . The normalized gradient at each point represents the directional guidance at that point in the workspace. Laplace equation for  2  is a Harmonic function.

17. NC STATE UNIVERSITY ADAC, NC State University17 Solving Laplace Equation Thus with Laplace equation, this method simply consists of repeatedly replacing each grid points with the average of its neighbors using successive relaxation. Terminate when the array u contains a sampling  of where every non-boundary condition node has a neighbor with a smaller value representing a negative gradient except the destination point. Using finite difference method, Taylor series approximation,

18. NC STATE UNIVERSITY ADAC, NC State University18 HPF with synthetic data The destination point is represented by the lowest potential (  = -1)

19. NC STATE UNIVERSITY ADAC, NC State University19 HPF and Edge detection E(x i, y j ) = 1 if (x i, y j )   B = 0 if (x i, y j )   B for all (i, j)  is the boundary of the obstacles Where E(x, y) is the edge map and  B is the set of edge points including the boundary points for all obstacles in workspace  B nothing but boundaries of the obstacle, , raised to a high potential. Thus it provides the exact raw data required for HPF in Dirichlet’s setting to create the gradient direction matrix. The destination point is then represented by the lowest potential (  = -1)

20. NC STATE UNIVERSITY ADAC, NC State University20 HPF, a region to point guidance function We observe that the HPF plan of the workspace is the function of obstacle boundaries and the destination point. Thus HPF converts the edge map into a “Region to Point Guidance Function”

21. NC STATE UNIVERSITY ADAC, NC State University21 Goal Seeking with HPF The important feature of HPF planner is to convert the edge map into a region to point guidance function. the problem is converted to a “goal seeking” problem from a “path tracking” problem.

22. NC STATE UNIVERSITY ADAC, NC State University22 HPF with Motion Controller The reference position for each current position is calculated from the gradient array of the HPF (  ).  L – discretized look-ahead distance. (ex, ey, e  ) the error vector is calculated for (x 0, y 0,  0 ) and (x R, y R ) y = A n x 2

23. NC STATE UNIVERSITY ADAC, NC State University23 Effect of look-ahead distance  L = 1, Network delay = 0.1s T = 27 seconds  L = 8, Network delay = 0.1s T = 16 seconds  L = 8, Network delay = 0.6 s The magnitude of the speed v and the turn-rate  is proportional to the distance d 0 between (x 0, y 0 ) and (x ref, y ref ) Small  L – UGV close to the path – Small distance error – more time Large  L – Less time – large distance error – probability to hit the obstacle

24. NC STATE UNIVERSITY ADAC, NC State University24 Dynamic look-ahead distance G D – grid size distance Workspace image resolution – (m x n) = 320 x 240 Workspace size – (x a x y a ) = (4m x 3m) High curvature point  small d 0, small  L The UGV runs slowly on the turn. Path is a straight line (low curvature point) large d0, large  L. UGV moves faster. Optimality between the path tracking accuracy and the time required to reach the goal. Look-ahead distance (d 0 ) is function of curvature. From the Quadratic curve controller y = A n x 2 Network delay = 0.3 s T = 24.3 s

25. NC STATE UNIVERSITY ADAC, NC State University25 Edge detection Vs Template matching Template matching l Need of templates for the obstacles l Shape and size restriction on these obstacles. l Knowledge about only the location of the obstacles l No knowledge about the actual boundaries of the obstacles l Many restrictive assumptions about the environment Edge Detection l No need of templates l No Shape and size restrictions on the obstacles l Capture a full representation of the environment l Boundaries of the obstacles are marked distinctively. l No restrictive assumptions regarding its contents.

26. NC STATE UNIVERSITY ADAC, NC State University26 Fast Marching Vs HPF Fast Marching with Motion controller l Source to destination Reference path l Path tracking l Reference point calculation does not have obstacle vicinity knowledge. l Computationally and Memory intensive algorithm. HPF with Motion controller l Region to point Reference array for the whole workspace. l Goal seeking l Reference point is calculated using the gradient of HPF planner, which always drive UGV away from obstacles. l Less computationally intensive (simple averaging operation)

27. NC STATE UNIVERSITY ADAC, NC State University27 Real time computation Np – Number of points on reference path At each sampling instance ti – Np real time checks on the path to find out the closest point and (x ref, y ref ) say n.ti – time required to reach the goal (n – number of loop iterations) Total (Np.n) real time computations Np   L   p  1 Each reference point (xref, yref) computation in real time will take  L addition operations. Total (  L.n) real time computations. Performace improvement factor  p =

28. NC STATE UNIVERSITY ADAC, NC State University28 Key Interface points Old structure of iSpace 1. After template matching, safety radius has to be decided before the points can be passed on to the fast marching method. 2. Fast marching method has to keep track of all the points using a binary tree, whether the point lies inside or outside the safety circle. 3. Reference point calculation in real time is mandatory due to path tracking problem. New structure of iSpace 1. The output from edge detection can be directly fed to HPF planner without preprocessing 2. Knowledge of boundaries (high potential) make it easy with successive relaxation to get the gradient array 3. Reference array calculation from the HPF planner makes it quick for the quadratic curve fitting controller. The new structure has efficient interfacing

29. NC STATE UNIVERSITY ADAC, NC State University29 Results - 1 For Comparison purpose, ideal path from source to destination was generated

30. NC STATE UNIVERSITY ADAC, NC State University30 Results - 2 Left – HPF planner and Right – fast marching planner for same grid size Image with a barrier separating two regions. (Blue dot is the source and red dot is the chosen goal.)

31. NC STATE UNIVERSITY ADAC, NC State University31 Conclusion l Edge detection, a model based HPF planner and network based quadratic Controller go hand in hand to create an efficient and delay- tolerant integrated navigation system. l More generality and flexibility to the UGV workspace environment. l A good edge map helps to build the correct HPF planner. l The gradient array calculation from HPF planner decreases the computational burden in real time making - more suitable for network- based control. l The Dirichlet’s setting keeps the robot path, as much as possible, away from the obstacles making it efficient even in heavily cluttered environments. l The combined effect of HPF and the quadratic curve controller display intelligent behavior such as no movement in case of goal unreachable problems. Thus the new iSpace structure suggested satisfies many requirement which are key for a network based integrated navigation system.

32. NC STATE UNIVERSITY ADAC, NC State University32 Future Research l Improvement on edge detection as reliable edge detection is the backbone of the new structure of iSpace. l Considering the dimensions of the robot before creating the HPF planner to take care of the safety margin around the wall. (Possible solutions - Dilation of the edge map) l Using HPF for velocity control for the UGV. l Dynamic obstacle avoidance for NBINS.

33. NC STATE UNIVERSITY ADAC, NC State University33