1. The Limits of Localization Using Signal Strength: A Comparative Study Eiman Elnahrawy, Xiaoyan Li, and Richard Martin Dept. of Computer Science, Rutgers University SECON, October 7 th 2004

2. Dept. of Computer Science, Rutgers University Sensors Everywhere Tracking, monitoring, geometric-based routing, … “Localization in indoor environments” Off-the-shelf radios Already used for communication No additional hardware

3. Dept. of Computer Science, Rutgers University Background: Radio-Based Localization Two strategies: [1] Classifiers (matching or learning): match RSS to existing fingerprints [(x,y),SS] [2] Propagation model: distance as a function of signal strength: Sj = k 0j + k 1j log Dj, Dj = sqrt((x-x j ) 2 -(y-y j ) 2 ) Dj = sqrt((x-x j ) 2 -(y-y j ) 2 ) [Bahl00, Ladd02, Roos02, Smailagic02, Youssef03, Krishnan04] [-80,-67,-50] RSS (x j,y j ) (x?,y?) [(x,y),s1,s2,s3]

4. Dept. of Computer Science, Rutgers University Major Question What is the best performance we can achieve in radio- based localization without any additional hardware? Ultrasound Very high frequency clocks Directional antenna Related Work: different technologies/algorithms Related Work: different technologies/algorithms [Want92, Priyantha00, Doherty01, Niculescue01, Savvides01, Shang03, He03, Hazas03, Lorincz04]

5. Dept. of Computer Science, Rutgers University Strategy Compared localization performance across a wide spectrum of representative algorithms 2 previous + 3 new “area-based” + variances = 11 algorithms Used combination of traditional and new metrics Ran extensive experiments across different environments (2 different buildings) using 802.11 Combined results with a prior small-scale evaluation on matching algorithms (classifiers) [Battiti02]

6. Dept. of Computer Science, Rutgers University Results in a Nutshell All algorithms have similar performance Fundamental limitations due to noise and systematic errors Area-based algorithms allow Precision-Accuracy tradeoffs Within existing HW and using simple floor maps we can get 95% room-accuracy Median error of 10 feet and 97 th percentile of 30 feet To improve, use additional hardware (e.g., directional antenna) or a more complex environment model (i.e., model every shelf, desk!)

7. Dept. of Computer Science, Rutgers University Outline Introduction, Motivations, and Related Work Localization Algorithms: An Overview Experimental Evaluation Performance metrics Experimental setup Results Conclusion and Future Work

8. Dept. of Computer Science, Rutgers University Point-based Versus Area-based “Point-based” algorithms Returned answer is a single location “Area-based” algorithms Returned answer is an area (or volume) likely to contain the localized sensor Ability to describe the uncertainty: possible loc Accuracy (likelihood of being in the area) vs. Precision (area size)

9. Dept. of Computer Science, Rutgers University Radio-Based Algorithms at a Glance (1) Simple Point Matching (SPM) (2) Area Based Probability (ABP) (3) Bayesian Networks (BN) (6) RADAR (7) Averaged RADAR (8) Gridded RADAR (9) Highest probability (10) Averaged highest probability (11) Gridded highest probability (4) Bayesian point (5) Averaged Bayesian MatchingSS vs. distance Area-based Point-based

10. Dept. of Computer Science, Rutgers University 1. SPM Find tiles which fall within a “threshold” of RSS for each AP Find ∩ tiles which fall within a “threshold” of RSS for each AP ∩∩ = 2. ABP-c Using “Bayes’ rule” compute likelihood of an RSS matching a fingerprint for each tile p(Ti|RSS) α p(RSS|Ti). p(Ti) Return a set of tiles bounded by an overall probability that the sensor lies in the area (Confidence: user-defined) Build a regular grid of tiles, tile ↔ expected fingerprint Eager: start from low threshold (= δ, 2 × δ, …) Similar to MLE Confidence ↑ → Area size ↑

11. Dept. of Computer Science, Rutgers University Measurement At Each Tile Is Expensive! Triangle-based linear interpolation using “Delaunay Triangulation(Surface Fitting) Triangle-based linear interpolation using “Delaunay Triangulation” (Surface Fitting) Simple, fast, and efficient Insensitive to the tile size

12. Dept. of Computer Science, Rutgers University 3. Bayesian Networks Graphical models that encode dependencies between variables Propagation model: Dj is the distance to the jth AP Sj = k 0j + k 1j log Dj Dj = sqrt((x-x j ) 2 -(y-y j ) 2 ) Dj = sqrt((x-x j ) 2 -(y-y j ) 2 ) Network learns parameters k 0j, k 1j, x, y distribution (joint PDF) using training fingerprints S1S1 S2S2 S3S3 S4S4 D1D1 D2D2 D3D3 D4D4 X Y

13. Dept. of Computer Science, Rutgers University Point Based Algorithms… (4) Bayesian Point, (5) Averaged Bayesian (6) RADAR Return the “closest” fingerprint to the RSS in the training set using “Euclidean Distance in signal space” (7) Averaged RADAR, (8) Gridded RADAR (9) Highest Probability Similar to ABP: a typical approach that uses “Bayes’ rule” but returns the “highest probability single location” (10) Averaged Highest Probability, (11) Gridded Highest Probability

14. Dept. of Computer Science, Rutgers University Performance Metrics Traditional: Distance error between returned and true position Return avg, 95 th percentile, or full CDF Does not apply to area-based algorithms! Does not show accuracy-precision tradeoffs!

15. Dept. of Computer Science, Rutgers University New Metrics: Accuracy Vs. Precision Tile Accuracy % true tile is returned Distance Accuracy distance between true tile and returned tiles (sort and use percentiles to capture distribution) Precision size of returned area (e.g., sq.ft.) or % floor size

16. Dept. of Computer Science, Rutgers University Room-Level Metrics Applications usually operate at the level of rooms Mapping: divide floor into rooms and map tiles (Point ↔ Room): easy (Area ↔ Room): tricky Metrics: accuracy-precision Room Accuracy % true room is the returned room Top-n rooms Accuracy % true room is among the returned rooms (for area-based algorithms) Room Precision avg number of returned rooms (for area- based algorithms)

17. Dept. of Computer Science, Rutgers University Experimental Setup CoRE 286 fingerprints (rooms + hallways) 50 rooms 200x80 feet 4 Access Points (somewhat linear) Industrial research lab 253 fingerprints (all hallways) 20 rooms out of 115 rooms 225x144 feet 5 Access Points (nicely distributed)

18. Dept. of Computer Science, Rutgers University [1] Impact of Training on Accuracy and Precision Divide data into training and testing sets Both location and number of training samples impact performance Different strategies [Fixed spacing vs. Average spacing]: as long as samples are “uniformly distributed” but not necessarily “uniformly spaced” methodology has no measurable effect

19. Dept. of Computer Science, Rutgers University Number of samples has an impact, but not strong! Improving Accuracy worsens Precision (tradeoff)

20. Dept. of Computer Science, Rutgers University 020406080100 0 0.2 0.4 0.6 0.8 1 probability ABP-75: Percentiles' CDF Minimum 25 Percentile Median 75 Percentile Maximum 020406080100 0 0.2 0.4 0.6 0.8 1 probability ABP-50: Percentiles' CDF Minimum 25 Percentile Median 75 Percentile Maximum 020406080100 0 0.2 0.4 0.6 0.8 1 distance in feet probability SPM: Percentiles' CDF Minimum 25 Percentile Median 75 Percentile Maximum 020406080100 0 0.2 0.4 0.6 0.8 1 distance in feet probability ABP-95: Percentiles' CDF Minimum 25 Percentile Median 75 Percentile Maximum [2] A Deeper Look Into “Accuracy”

21. Dept. of Computer Science, Rutgers University [3] Comparing All Algorithms: Accuracy Traditional error along with median CDF for area- based algorithms Room-level accuracy

22. Dept. of Computer Science, Rutgers University 020406080100 0 0.2 0.4 0.6 0.8 1 distance in feet probability Error CDF Across Algorithms BN-Median ABP-75 Median R1 R2 GR P1 P2 GP B1 B2 Striking similarity! Similar slope, percentiles Increasing sample size yields marginal improvement Given [Battiti et al] compared a host of “learning approaches” including MLE and found similar performance » ”all algo are qualitatively similar”

23. Dept. of Computer Science, Rutgers University Algorithm Probability Within Room Accuracy Stack SA50A75A95BNR1R2GRP1P2GPB1B2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 First Second Third Others Similar top-room accuracy Area-based algorithms are superior at returning multiple rooms Again, increasing sample size yields marginal improvement

24. Dept. of Computer Science, Rutgers University [4] Fundamental Uncertainty? Bayesian net used to generate spatial uncertainty PDF across x,y Wide distributions » high degree of uncertainty Attenuation along x in CoRE helps reducing uncertainty 050100150200 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 x in feet probability 01020304050607080 0 0.02 0.04 0.06 0.08 0.1 0.12 y in feet probability 050100150200 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 x in feet probability 020406080100120140 0 0.01 0.02 0.03 0.04 0.05 0.06 y in feet probability CoREResearch Lab

25. Dept. of Computer Science, Rutgers University Take Home Lessons All algorithms behave the same: there is a fundamental uncertainty due to environmental effects! Point-based approaches find the peaks in the PDFs Area-based approaches explore more of the PDF but cannot narrow it (cannot eliminate the uncertainty) Area-based approaches offer a precision-accuracy tradeoff Uniform distribution with sufficient density: a rule of thumb Sample density of 1/230ft 2 (every 15 ft) Reasonable performance at 1/450ft 2 (every 21 ft)

26. Dept. of Computer Science, Rutgers University Conclusion and Future Work Characterized the limits of a wide variety of approaches to localization using 802.11 Algorithms based on matching and signal-to-distance functions are unable to capture effects on signal propagation Useful accuracy, different ways to describe it to users Uncertainty cannot be reduced! Additional HW or complex models to improve accuracy Future work: hmm! Would it be worth the improvement

27. Dept. of Computer Science, Rutgers University Thank You