1. EvAAL 2012 - Evaluating AAL Systems through Competitive Benchmarking
A balance of the 2nd edition
Stefano Chessa
EvAAL final workshop
24/09/2012

2. Motivations
Ambient Assisted Living (AAL) seeks for technical solutions for the improvement of the quality of life of elders and disabled.
EvAAL aims at giving a pulse to research in AAL:
by identifying research challenges relevant to AAL
The long term research challenge of EvAAL is the evaluation of complex platforms for AAL
Short term research challenges are the evaluation of components/services of AAL systems
by making available to researchers the datasets obtained during the competition;
by contributing to the construction of a community of people interested in AAL.
EvAAL final workshop
24/09/2012

3. EvAAL : a brief history and current status
EvAAL is a result of an EU project (universAAL)
It was conceived in early 2010
In 2011 the first edition
Only one track on Indoor Localization and Tracking for AAL
Relatively easy to formalize as a competition
In 2012 the second edition
An additional track on Activity Recognition for AAL
Growing community
Already planning the 2013 edition
With (hopefully) another track
EvAAL final workshop
24/09/2012

4. The EvAAL 2012 team Steering board (9 members)
Italy (3), Spain (3) Germany (2) Denmark (1)
Program Committee of the Indoor Localization and Tracking track
19 members from EU, USA and Australia
The local and evaluation team
EvAAL final workshop
24/09/2012

5. The EvAAL 2012 team
Program Committee of the Activity Recognition track
16 members from EU, USA and Japan (17 members from Spain and Italy)
Local and evaluation teams
EvAAL final workshop
24/09/2012

6. Competitors selection process
Indoor Localization: 9 submissions
Each submission was reviewed by 3 TPC members
1 rejected as not suitable for AAL
1 admitted as guest
7 admitted as competitors
Activity Recognition: 6 submissions
Each submission was reviewed by 2 TPC members
1 withdrew
4 admitted as competitors
EvAAL final workshop
24/09/2012

7. Indoor Localization Track
iLocPlus (Germany/Switzerland)
CPS Utah (USA)
CAR (Spain)
LOCOSmotion (Germany)
Grupo TAIS (Spain)
OwlPS (France)
Smart-Condo (Canada)
Lambda4 (Germany) - GUEST
EvAAL final workshop
24/09/2012

8. Activity Recognition Track
SenseCam (Ireland)
AMEVA (Spain)
Chiba University (Japan)
Carnegie Mellon University (USA)
EvAAL final workshop
24/09/2012

9. The Localization Benchmark
The benchmark consists of a set of tests, each of which contributes to assessment of the scores for the competing system.
Phase 1: Area of interest test
Phase 2: Locate the actor
Phase 3: The disturber
Each localization system need to produce localization data with a frequency of 1 data every half a second.
The paths followed by the person were the same for each test, and it was not disclosed to competitors before the competition.
We also provide some context information
Switch
Cyclette
EvAAL final workshop
24/09/2012

10. Area of Interest
Phase 1: Locate a person inside an Area of Interest (AoI).
The AoI in a typically AAL scenario could be inside a specific room (bathroom, bedroom), in front of a kitchen etc.
Each system was requested to identify:
6 big AoI
4 hierarchical small AoI.
The actor moved along predefined paths staying inside each AoI for at least seconds (227 steps)
Starting Point
End Point
EvAAL final workshop
24/09/2012

11. The Actor
Phase 2: A person that moves inside the Living Lab had to be located and tracked.
Only the person to be localized was inside the Living Lab.
The first path was 54 steps length, the second path 94.
All the paths were characterized by 3 waiting points, i.e. the actor stayed in the same position for 5 seconds.
EvAAL final workshop
24/09/2012

12. The disturber
Phase 3: Was organized as the former phase, but the competing artefacts were evaluated in the presence of another person who moves inside the Living Lab.
Only the actor had to be localized by the competing artefacts
There was no requirement of localizing the disturber
Only the actor has carried equipment provided by the competitor
The actor started moving at least 5 seconds before the disturber
Both the actor and the disturber have generated contextual events but the first such event was generated by the actor
When the disturber generated an event, the actor was at least 2 meters away.
The first path was 94 steps length (while the disturber path was 54), the second path 80 (while the disturber path was 94).
EvAAL final workshop
24/09/2012

13. The reference localization system
The reference localization system is essential to measure the accuracy of the competing systems.
Was composed by predetermined coordinates of the paths followed by the actor during the competition
The Living Lab’s floor was covered with marks (with different color to distinguish the right and left foot) that show where the actor had to step on.
In order to facilitate the installation and removal of the paths the marks were put on a piece of wood.
The synchronization between the steps and the evaluation tool was guaranteed by a digital metronome that indicated the right cadence (one beep one step).
In this way we guaranteed that the actor repeated the same paths at the same speed for every competitor.
EvAAL final workshop
24/09/2012

14. Evaluation criteria Accuracy [weight 0.25] Availability [weight 0.2]
This is the main purpose of a localization system
The score is attributed in a range of errors between 0 and 4 meters
An average error above 4 meters would mean that the system does not really know in which room the user is…
Availability [weight 0.2]
…nobody wants a system that does not responds…
EvAAL final workshop
24/09/2012

15. Evaluation criteria Installation complexity [weight 0.15]
Measure the invasiveness of the system installation in the home of the final user
Think whether if you would be happy to install these systems in your home…
Interoperability with AAL systems [weight 0.15]
How easy is to use your system from the point of view of AAL applications programmers
EvAAL final workshop
24/09/2012

16. Evaluation criteria User acceptance [weight 0.25]
Would you use your system in your daily life?
More advanced and better engineered systems where advantaged against research prototypes…
… but we considered also your system in the perspective of an engineering
EvAAL final workshop
24/09/2012

17. Financing of EvAAL All the organization members were volunteers
The Smart House Living Lab and the CIAMI Living Lab were offered for free
EvAAL final workshop
24/09/2012

18. SPONSORS
ASUS and Texas Instruments contributed with prizes for winners
GIRAFF Plus and universAAL contributed with free registrations to AAL Forum for competitors
EvAAL final workshop
24/09/2012