1. The EVARILOS benchmarking methodology
16/09/2018
The EVARILOS benchmarking methodology
Evaluation of RF-based Indoor Localization Solutions
Ingrid Moerman
16/09/2018

2. Introduction – indoor navigation
16/09/2018
Introduction – indoor navigation
Navigation in huge complex buildings
Advanced nurse calling systems
Tracking of patients with Alzheimer's
Underground mining evacuation and navigation
Indoor navigation is these days a hot topic in a lot of different research areas. It is not only useful for navigating in complex buildings, but also in eHealth is indoor localization very productive in order to optimize the nurse calling systems or for tracking patients with alzheimer’s. Another example is in the underground mining where indoor localization can save lives. In mines, sensor networks are used to measure and control the environment, but they can also be used to track mineworkers and help them evacuate when something is wrong.
16/09/2018

3. Introduction – problem statement
16/09/2018
Introduction – problem statement
A lot of different techniques: RSSI, ToA, TDoA, AoA, DTDoA, Proximity, …
A lot of different technologies: IEEE , RFID, IEEE , UWB, 60 GHz,…
Problem? Solutions are not comparable
Different building specifications Type of walls, number of nodes, …
Different interference specifications Other signals in the environment
Different priorities, needs Accuracy, response time, …
Because indoor localization becomes a hot topic, a lot of different techniques have been developed to locate a mobile node.
But the results published in papers are not comparable. Every solution is tested in a different environment with different building and interference specifications.
The environment can drastically change the final results. Besides, in some cases the priority can be different, sometimes the accuracy is very important but in other cases the response time might be an important factor.
So the conclusion is, we can not compare apples to oranges (blijkbaar de engelse versie van “appels met peren vergelijken”).
A structured, well defined, benchmarking methodology is necessary in order to evaluate these indoor localization solutions.
16/09/2018

4. Introduction - example
Metric / Technology
RSSI & Radio
ToA & UWB
Point accuracy
+/- ++
Room accuracy +
Energy efficiency --
Price per unit
Two totally different techniques & technologies
Based on accuracy  Option 2 is the best solution
However, based on all metrics  Option 1 is mostly the ‘best’
Requirements are different for every situation
 Urgent need for fair and advanced comparison
16/09/2018

5. EVARILOS benchmarking methodology
16/09/2018
EVARILOS benchmarking methodology
Goals:
Objective experimental validation
Fair comparison
Wide range of metrics
Improve interference robustness
Testing in real-life applications
Results:
Public handbook
Software suite
Goals:
EVARILOS will develop a benchmarking methodology enabling objective experimental validation of and fair comparison between state-of-the art indoor localization solutions.
The EVARILOS benchmarking methodology not only considers accuracy metrics, but also complexity, cost, energy, and, most importantly, RF interference robustness metrics.
The project aims to improve the interference robustness of state-of-the art localization solutions using multimodal approaches or cognitive radios and environmental awareness,…
This will be valuated in two real-life scenarios: a healthcare hospital setting and underground mining safety.
The main outcomes of the project are a public handbook on the use of the EVARILOS benchmarking methodology and the EVARILOS benchmarking suite.
16/09/2018

6. EVARILOS handbook Methodology: defining scenarios Example 7.7/10
16/09/2018
EVARILOS handbook
Methodology: defining scenarios
Example
Solution using RSSI
and fingerprinting
Type of wall: brick
Size: big
“Office interference”
WiFi, bluetooth, 3G,…
Min 10 measurements
close to wall
Relevant metrics:
Accuracy and latency
The main chapter in the EVARILOS benchmarking handbook is the benchmarking methodology. In this chapter, all the essential parts of a scenario are explained and defined.
The scenario begins with a given localization solution that will be tested. Then, additional information is required to have a correct setup: environment specifications (containing the building and interference specifications), the measurement points specifications (location of the mobile node, the frequency of measurement, the amount,…) and most important a selection of metrics.
All evaluations are considered black box benchmarks: the scenario description can be seen as a black box, which takes as input a localization approach, and outputs one or more numerical benchmarking values. As such, the internal properties of the localization
scenario are not evaluated, only its relevance for different application domains.
A very simple example is given. Suppose we have a solution that we would like to evaluate that uses RSSI measurements and requires fingerprinting. The selected testbed will determine the environment specifications. In our example, the testbed contains brick walls and belongs to the category “Big”. In the EVARILOS handbook, all these specifications are clearly defined. This also applies to the interference specifications. In our example, the standard office interference is chosen whereby wifi, bluetooth and 3G are used. Further, the measurement point specifications are some restrictions about the measurements itself, the amount, where, the frequency, etc. In our example minimum 10 measurements need to be close to a wall, there are, depending of the scenario, these restrictions can change of course.
Next, a set of metrics is chosen that are relevant for this scenario, in our case the accuracy and latency, more about the metrics will be explained further in this presentation.
All these settings are all part of the setup. Thereafter, the evaluation criteria must be defined, which metric is important and how is the final score calculated. In our case the accuracy is much more important than the latency, so the weight factor of the accuracy is much higher. Finally, in the conclusion we can summarize and evaluate the results of the scenario.
Weight factors:
Accuracy: 0.8
Latency: 0.2
7.7/10
Scores:
Accuracy: 8/10
Latency: 6.5/10
16/09/2018

7. EVARILOS handbook Methodology: defining scenarios 16/09/2018
In the following part of this presentation, the metrics will be more explained in detail because they have an important impact on the evaluation of the localization solutions.
16/09/2018

8. The EVARILOS metrics Metric: Three categories: Definition
16/09/2018
The EVARILOS metrics
Metric:
Definition
Measurement method
Mathematical equation
Three categories:
Performance metrics (primary and derived)
Functional metrics
Deployment metrics
A metric is a measure of a specific performance indicator of the system under test. In our handbook a metric contains always a definition, instructions for the underlying measurements and the mathematical equation that would be used in order to calculate the metric value.
Not only accuracy or other performance parameters are used. The EVARILOS methodology takes into account the multifaceted nature of localization schemes. Therefore, three different categories of metrics are defined: the performance, functional and deployment metrics.
An overview is given on the next slide.
16/09/2018

9. The EVARILOS metrics 16/09/2018
This is an overview of all the metrics. A noticeable remark is that there are derived performance metrics. These metrics conclude something about the performance of the solution, but one of the primary performance metrics are necessary in order to evaluate the derived performance metrics.
In the next slides, point accuracy and interference robustness will be explained as example.
16/09/2018

10. Metrics: point accuracy
16/09/2018
Metrics: point accuracy
Definition: Euclidian error distance between a reference and measurement point.
Method: Euclidian distance equation with 3D coordinates:
The description of point accuracy is the Euclidian error distance between a reference and measurement point. The coordinates can be 2D or 3D, in this example, the 3D coordinates are used.
Of course, these measurements are repeated multiple times, depending on the restrictions of the scenario. In order to have a quick overview of the measurements, the average, standard deviation, minimum and maximum values are also calculated.
16/09/2018

11. Metrics: interference robustness
16/09/2018
Metrics: interference robustness
Definition: Comparison of performance metrics under different circumstances (with and without interference)
Method (2 phases):
Study of the (in this case) accuracy with interference (same equations as the previous slide)
Comparison with the accuracy without interference
An example of the derived performance metrics is the interference robustness. This metric will compare the performance results under different circumstances.
To measure this metric, 2 phases are necessary. The first phase is to evaluate the performance metric (in our example the accuracy) with the interference in the environment. This is on the assumption that the results of the accuracy metric are already available without interference.
Then, the second phase is the comparison between both results. The final scores are the differences between both results.
16/09/2018

12. Interference robustness
16/09/2018
Interference robustness
Adding new class of approaches
Combat the interference drawbacks
Environmental awareness (cognitive radios)
Evaluation with our methodology
The main purpose: perform well in real-life Future Internet environments
EVARILOS also has a second goal, the development and evaluation of localization solutions that add RF interference robustness to (existing) indoor localization solutions.
We will try to add a new class of approaches to combat the interference drawbacks.
This automatically linked to environmental awareness and coexistence with other users and/or technologies. This goes hand in hand with cognitive radios and dynamic spectrum access.
This solutions will also be evaluated using the EVARILOS benchmarking methodology described in the previous slides.
The main purpose of this second part is to create indoor localization solutions that perform well in real-life Future Internet environments.
16/09/2018

13. The EVARILOS project Main purposes Part I: the benchmark methodology
16/09/2018
The EVARILOS project
Main purposes
Part I: the benchmark methodology
Handbook (with scenarios and metrics)
Software suite (for the open call)
Part II: Interference robustness
Developing new class of approaches
Evaluating our solutions with the benchmark methodology
Testing the solutions in 2 real-life environments
Healthcare environment (hospital, elderly home)
Underground mine
We can summarize the EVARILOS project in two main parts.
The first part is the benchmark methodology that contains the handbook, this is including all the defined metrics and scenarios, and the software suite that will be used for the open call.
The second part handles about interference robustness and environmental awareness whereby a new class of approaches will be developed to combat the interference drawbacks.
These solutions will be evaluated with our own methodology from the first part.
16/09/2018

14. The EVARILOS project Consortium:
16/09/2018
The EVARILOS project
Consortium:
Technische Universität Berlin (coordinator)
Advantic Sistemas y Servicios
iMinds
Swedisch Institute for Computer Science
Televic Healthcare
Duration: 26 months (nov 2012 – jan 2015)
Two different testbed locations
TWIST testbed Berlin (TUB)
w-iLab.t in Ghent and w-iLab.t II in Zwijnaarde (iMinds)
The EVARILOS project contains 5 different partners, 3 academic partners (TUB, SICS and iMinds) and two industrial partners (THC and ADV).
The projects duration is 26 months and is already started since November 2012.
To test and evaluate the solutions and our methodology, two different testbed locations are available.
The TWIST testbed in Berlin and the two w-iLab.t testbeds in Ghent and Zwijnaarde.
16/09/2018

15. The EVARILOS testbeds TWIST Berlin w-iLab.t Ghent
w-iLab.t II Zwijnaarde
16/09/2018