1. A BALANCE OF THE 2 ND EDITION STEFANO CHESSA EvAAL 2012 - Evaluating AAL Systems through Competitive Benchmarking 1 EvAAL 2012 - final workshop 24/09/2012

2. Motivations 24/09/2012 EvAAL 2012 - final workshop 2 Ambient Assisted Living (AAL) seeks for technical solutions for the improvement of the quality of life of elders and disabled. EvAAL aims at giving a pulse to research in AAL:  by identifying research challenges relevant to AAL  The long term research challenge of EvAAL is the evaluation of complex platforms for AAL  Short term research challenges are the evaluation of components/services of AAL systems  by making available to researchers the datasets obtained during the competition;  by contributing to the construction of a community of people interested in AAL.

3. EvAAL : a brief history and current status 24/09/2012 EvAAL 2012 - final workshop 3 EvAAL is a result of an EU project (universAAL) It was conceived in early 2010 In 2011 the first edition  Only one track on Indoor Localization and Tracking for AAL  Relatively easy to formalize as a competition In 2012 the second edition  An additional track on Activity Recognition for AAL  Growing community Already planning the 2013 edition  With (hopefully) another track

4. The EvAAL 2012 team 24/09/2012 EvAAL 2012 - final workshop 4 Steering board (9 members)  Italy (3), Spain (3) Germany (2) Denmark (1) Program Committee of the Indoor Localization and Tracking track  19 members from EU, USA and Australia The local and evaluation team

5. The EvAAL 2012 team 24/09/2012 EvAAL 2012 - final workshop 5 Program Committee of the Activity Recognition track  16 members from EU, USA and Japan (17 members from Spain and Italy) Local and evaluation teams

6. Competitors selection process 24/09/2012 EvAAL 2012 - final workshop 6 Indoor Localization: 9 submissions  Each submission was reviewed by 3 TPC members  1 rejected as not suitable for AAL  1 admitted as guest  7 admitted as competitors Activity Recognition: 6 submissions  Each submission was reviewed by 2 TPC members  1 rejected as not suitable for AAL  1 withdrew  4 admitted as competitors

7. Indoor Localization Track iLocPlus (Germany/Switzerland) CPS Group @ Utah (USA) CAR (Spain) LOCOSmotion (Germany) Grupo TAIS (Spain) OwlPS (France) Smart-Condo (Canada) Lambda4 (Germany) - GUEST 7 24/09/2012 EvAAL 2012 - final workshop

8. Activity Recognition Track SenseCam (Ireland) AMEVA (Spain) Chiba University (Japan) Carnegie Mellon University (USA) 8 24/09/2012 EvAAL 2012 - final workshop

9. The Localization Benchmark 24/09/2012 EvAAL 2012 - final workshop 9 The benchmark consists of a set of tests, each of which contributes to assessment of the scores for the competing system.  Phase 1: Area of interest test  Phase 2: Locate the actor  Phase 3: The disturber Each localization system need to produce localization data with a frequency of 1 data every half a second. The paths followed by the person were the same for each test, and it was not disclosed to competitors before the competition. We also provide some context information  Switch  Cyclette

10. Area of Interest 24/09/2012 EvAAL 2012 - final workshop 10 Phase 1: Locate a person inside an Area of Interest (AoI).  The AoI in a typically AAL scenario could be inside a specific room (bathroom, bedroom), in front of a kitchen etc.  Each system was requested to identify:  6 big AoI  4 hierarchical small AoI.  The actor moved along predefined paths staying inside each AoI for at least seconds (227 steps) Starting Point End Point

11. The Actor 24/09/2012 EvAAL 2012 - final workshop 11 Phase 2: A person that moves inside the Living Lab had to be located and tracked.  Only the person to be localized was inside the Living Lab.  The first path was 54 steps length, the second path 94.  All the paths were characterized by 3 waiting points, i.e. the actor stayed in the same position for 5 seconds.

12. The disturber 24/09/2012 EvAAL 2012 - final workshop 12 Phase 3: Was organized as the former phase, but the competing artefacts were evaluated in the presence of another person who moves inside the Living Lab.  Only the actor had to be localized by the competing artefacts  There was no requirement of localizing the disturber  Only the actor has carried equipment provided by the competitor  The actor started moving at least 5 seconds before the disturber  Both the actor and the disturber have generated contextual events but the first such event was generated by the actor  When the disturber generated an event, the actor was at least 2 meters away.  The first path was 94 steps length (while the disturber path was 54), the second path 80 (while the disturber path was 94).

13. The reference localization system 24/09/2012 EvAAL 2012 - final workshop 13 The reference localization system is essential to measure the accuracy of the competing systems.  Was composed by predetermined coordinates of the paths followed by the actor during the competition  The Living Lab’s floor was covered with marks (with different color to distinguish the right and left foot) that show where the actor had to step on.  In order to facilitate the installation and removal of the paths the marks were put on a piece of wood. The synchronization between the steps and the evaluation tool was guaranteed by a digital metronome that indicated the right cadence (one beep one step).  In this way we guaranteed that the actor repeated the same paths at the same speed for every competitor.

14. Evaluation criteria 24/09/2012 EvAAL 2012 - final workshop 14 Accuracy [weight 0.25]  This is the main purpose of a localization system  The score is attributed in a range of errors between 0 and 4 meters  An average error above 4 meters would mean that the system does not really know in which room the user is… Availability [weight 0.2]  …nobody wants a system that does not responds…

15. Evaluation criteria 24/09/2012 EvAAL 2012 - final workshop 15 Installation complexity [weight 0.15]  Measure the invasiveness of the system installation in the home of the final user  Think whether if you would be happy to install these systems in your home… Interoperability with AAL systems [weight 0.15]  How easy is to use your system from the point of view of AAL applications programmers

16. Evaluation criteria 24/09/2012 EvAAL 2012 - final workshop 16 User acceptance [weight 0.25]  Would you use your system in your daily life?  More advanced and better engineered systems where advantaged against research prototypes…  … but we considered also your system in the perspective of an engineering

17. Financing of EvAAL 24/09/2012 EvAAL 2012 - final workshop 17

18. SPONSORS ASUS and Texas Instruments contributed with prizes for winners GIRAFF Plus and universAAL contributed with free registrations to AAL Forum for competitors 18 24/09/2012 EvAAL 2012 - final workshop