Feasibility of Interactive Localization and Navigation of People with Visual Impairments Ilias Apostolopoulos, Navid Fallah, Eelke Folmer and Kostas E. Bekris Computer Science and Engineering 1 September 2010, IAS

Introduction Problem Navigation in an indoor environment for individuals with visual impairment(VI) Solution Create a navigation system using: Minimal sensors (i.e. compass, pedometer) available in smart phones Interaction with the user Techniques from robotics

Wall Hallway intersection Door

Motivation Outdoors Outdoor navigation systems typically use GPS for localization Indoors Indoor navigation systems use RFID tags, cameras, laser scanners Individuals with VI navigate with compensatory senses (e.g., touch) -Results in reduced mobility A need for navigation assistance rises Easy Challenging

Motivation Indoor navigation should be accurate and efficient Solution proposed here requires only minimal sensors that can be found in a smart phone. Easier to create virtual infrastructure instead of a physical one

Characteristics of the Approach User interacts with the phone through an audio/speech interface Phone provides directions using landmarks that are recognizable by individuals with VI(i.e. doors, hallway intersections) User confirms landmarks through touch Directions Audio Feedback Speech Landmark Confirmation Desired Destination

Challenges and Premises Challenges Is there enough computational power on widely available portable devices? Is the sensing information sufficient? Can people perform well as sensors in this setup? Premises Individuals with VI can recognize landmarks through touch Indoor spaces, while complex, are highly constrained While pedometers and compasses are highly erroneous: They can provide a sufficiently good estimate for the user’s motion When integrated with state of the art methods for localization

Objectives of Feasibility Study Two research questions 1.Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided affect the probability of success? 2.Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

Background

Localization Techniques 1.Dead-Reckoning Integrate measurements of the human motion Accelerometers and radars have been used -P. Van Dorp et al., Human walking estimation with radar, 2003 Error grows unbounded 2.Beacon-based Use identifiers in the physical space Beacons can be detected by cameras, infrared or ultrasound identifiers Popular solutions use RFID tags -V. Kulyukin et al., Rfid in robot-assisted indoor navigation for the visually impaired, 2004 Locating beacons might be hard and inaccurate Significant time and cost spent installing and calibrating beacons

Localization Techniques 3.Sensor-based Use cameras to detect pre-existing features (e.g., walls, doors) -Cameras need good lighting and have prohibitive computational cost -O. Koch and S. Teller, Wide-area egomotion estimation from known 3d structure, 2007 Use 2D laser scanners -Expensive and heavy -J. A. Hesch et al., An indoor localization aid for the visually impaired, 2007 The system proposed is a sensor-based system Pedometer and compass for localization User as a sensor for landmark confirmation Use of probabilistic tools from robotics

Bayesian Methods from Robotics Candidate methods: 1.Extended Kalman Filter Assumes normal distribution +Returns optimum estimate under certain assumptions -Not a good model for multimodal distribution R. E. Kalman, A new approach to linear filtering and prediction problems, Particle Filter Employs a population of discrete estimates +Can represent a multimodal distribution -High computational cost N. J. Gordon et al., “Novel approach to nonlinear/non-gaussian bayesian state estimation”, 1993 Particle Filter is chosen due to better accuracy about the location of the user

Methodology Part 1: Directions

High-level Operation 1.A user specifies a start and destination room number to travel to. 2.The system computes the shortest path using A* and finds landmarks along the path. 3.Directions are provided iteratively upon completion through the phone’s built-in speaker. The user presses a button on the phone after successfully executing each direction.

Types of Directions The type of directions affects the efficiency and reliability of navigation High reliability when user confirms all landmarks, but low efficiency High efficiency when relying more on odometry, but low reliability Two types of direction provision were tested Landmark based Metric based Three granularities for each method No Max Threshold 15 Meters Threshold 9 Meters Threshold Metric based Landmark based Intersection Door 15 steps 20 steps

Types of Directions Metric based Intersection Door 15 steps 20 steps Landmark based Directions Metric based Landmark based No Max Threshold 9 Meters Threshold 15 Meters Threshold No Max Threshold 9 Meters Threshold 15 Meters Threshold

Direction Provision Landmark No Max Threshold "Exit the room then turn right" "Move forward until you reach a hallway on your left“ "Turn left to the hallway“ "Move forward until you reach a water cooler on your left“ "Move forward until you reach a hallway on your left“ "Turn left to the hallway" "Follow the wall on your left until you reach the third door“ "You have reached your destination"

Direction Provision Landmark 15 Meters Threshold "Exit the room then turn right“ "Move forward until you reach a hallway on your left“ "Turn left to the hallway“ "Follow the wall on your left until you reach the third door“ "Move forward until you reach a water cooler on your left" "Move forward until you reach a hallway on your left" "Turn left to the hallway" "Follow the wall on your right until you reach the 5th door" "Follow the wall on your left until you reach the first door" "You have reached your destination"

Direction Provision Landmark 9 Meters Threshold "Exit the room then turn right" "Follow the wall on your right until you reach the first door" "Move forward until you reach a hallway on your left“ "Turn left to the hallway" "Follow the wall on your left until you reach the second door“ "Move forward until you reach a water cooler on your left“ "Move forward until you reach a hallway on your left" "Turn left to the hallway" "Follow the wall on your right until you reach the third door“ "Follow the wall on your left until you reach the first door" "You have reached your destination"

Direction Provision Metric No Max Threshold "Exit the room then turn right" "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "You have reached your destination"

Direction Provision Metric 15 Meters Threshold "Exit the room then turn right“ "Walk x steps until you reach a hallway on your left" "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a door on your left" "You have reached your destination"

Direction Provision Metric 9 Meters Threshold "Exit the room then turn right"; "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway"; "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a door on your left“ "You have reached your destination"

Methodology Part 2: Localization

Localization Objective is an accurate location of the user Previous location and sensor data are used as input The new location is the output of the system Model n static landmarks ξ = (x, y, θ) state of the system m map that stores information about the world Landmarks belong to k different types (e.g, doors, hallway intersections)  Landmarks in the same class are indistinguishable Data d T = (o(0:T), u(0:T-1))  u(0:T-1) transitions  o(0:T) observations

Transitions and Observations Transitions A transition corresponds to a motion where the agent acquires the orientation and moves forward This transition determines the kinematic transition model of the system Observations Observation of a landmark from state implies: (x t+1,y t+1,θ t+1 ) utfutf utθutθ (x t,y t,θ t ) (x t, y t ) R obs lili (x i, y i )

Transitions and Observations Transitions A transition corresponds to a motion where the agent acquires the orientation and moves forward Observations Observation of a landmark from state implies: (x t+1,y t+1,θ t+1 ) utfutf utθutθ (x t,y t,θ t ) (x t, y t ) R obs lili (x i, y i )

Filtering The objective is to be able to incrementally estimate the user’s state at time T. The general Bayes filter computes a belief distribution at time T over given data d T. It requires:  An initialization B 0  A transition model  The observation model Given the normalization factor η the belief distribution can be updated as follows: Previous Belief Transition model Observation model Applying Transition model filter with observation

Filtering The objective is to be able to incrementally estimate the user’s state at time T. Given the normalization factor η the belief distribution can be updated as follows: Previous Belief Transition model Observation model Applying Transition model filter with observation

Particle Filter It is possible to represent B T through a set P of particles Each particle stores a state estimate together with a weight At each time step T, the following steps are: A.For each particle i.Employ the transition model to acquire ii.Employ the observation model to compute the new weight B.Sample a new population of P particles given the weights

Implementation of Transition Model Collects orientation from compass and step counts from pedometer Multiple readings from compass are averaged for a time step Pedometer returns 1 if the user moved a step, else 0 Step length is calculated for each user through some train paths Noise is added with a normal distribution to introduce uncertainty

Implementation of Observation Model Two cases: No landmark detected by user All particles get weight 1 Landmark detected Particles within observation range get weight inversely proportional to distance Particles out of range get weight 0

Sampling Algorithm samples with higher probability particles with higher weight When all particles get a weight of 0: Algorithm finds the closest visible landmark of the type observed by the user for each particle Particle is sampled near the landmark

Experiments

Setup System was implemented in Java for Google’s Android platform Map of a buildings floor in campus was created in Keyhole Markup Language(KML) Map contains: 3 water coolers 1 floor transition marked by a metal strip 3 hallway intersections & 2 hallway turns 72 doors 5 paths were defined two alternatives for directions with three granularities for each

Participants 10 volunteers Users held the phone in one hand and a cane in the other 1 volunteer was legally blind and assisted with the setup of the experiments 9 more sighted volunteers were blindfolded Some users had prior knowledge of the building while others didn’t Each user executed ten traversals Two traversals per path using different directions

Ground Truth An observer was recording the user’s motion Markers were placed on the floor every two meters Every time the user was crossing a marker the observer was recording the time in a second smart phone Assume that user moves with constant speed between markers Ground truth resolution 2 meters

Objectives of Feasibility Study Two research questions 1.Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided effect the probability of success? 2.Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

1.Success Ratio of Direction Provision 84% of experiments reached a distance smaller than 2 meters 92% of experiments reached less than 3.5 meters Distance from goal

1.Success Ratio of Direction Provision Duration of execution

2.Localization Accuracy Particle filter improves considerably results acquired just by sensor readings Paths with distinctive landmarks had considerably lower error than paths with repetitive landmarks Distance from Ground Truth

Localization Error Average error: 3.69 meters 2.Localization Accuracy

Discussion Proposed a minimalistic indoor navigation system for individuals with VI Sensors used are inexpensive and common though erroneous Answers to research questions: It is possible for an individual with VI to be guided successfully with this scheme It is possible to localize the user with accuracy

Update Submitted versionCurrent version Average error: 3.69 metersAverage error: 2.05 meters

Future Work Automatic direction provision based on localization estimates User studies with a larger number of users with VI More complex environments Buildings with multiple floors Elevators, stairs, ramps 3D maps with more details about the world

Thank you for your attention!