1. WP3 INERTIA Local Control and Automation Hub
Task 3.4 Multi-Sensorial Activity Flow Detection & Modelling in controlled environments
CERTH-ITI

2. Overview Task Progress / Main Achievements
Occupancy Extraction
New methods applied (CRF)
Sensor pre-editing of raw data at Arduino level
First results from deployed sensors and real data from CERTH
Individual Extraction (RFID)
Occupancy Prediction
Extension to current prototype development (added scheduling, etc.)
Validation against collected data
Integration with other components
Linksmart (CNET) & IAM (TUK)
Status of Deliverable D3.3
Next Efforts

3. Task Progress / Main Achievements Occupancy Extraction
Implemented new occupancy extraction algorithm based on Conditional Random Fields (CRF)
Using CRF is a new approach in real-time occupancy extraction (algorithm currently used in activity recognition, image segmentation, sentence analysis etc.)
Conditional Random Fields (CRF): Discriminative probabilistic model alternative to Hidden Markov Models (HMMs): Generative probabilistic models
CRF implementation compared to previous HMM approach
HMMs drawbacks:
ignore long term dependencies for:
hidden states (current hidden state depends only on the previous one)
observations
Strict independence assumptions on the observations
CRFs
do not make these assumptions, as they are undirected graphical models

4. Occupancy Extraction - CRF Algorithmic approach
Inputs:
Real-time sensor events produced by
installed multi-sensorial cloud (e.g. motion detectors, acoustic sensors, door counters etc.)
from monitored spaces (e.g. meeting room)
Trained CRF model for respective monitored space
Pre-Processing Component:
Handles sensor events
Prepares the input for the CRF:
Time is divided in timesteps of constant length
Computes the CRF features for each timestep, based on different parameters and passes them to CRF
CRF extracts a label (occupancy) for each timestep (currently configured at 10 secs).
Handles the output from CRF component
Output:
Estimated real-time occupancy per monitored space
Occupancy_Change events published to Linksmart

5. Occupancy Extraction – Arduino-based sensors integration
Door Counting (occupants entering/leaving through doors)
Integration and raw-data pre-processing at Arduino level:
Double-beam sensors:
The Break and Reset times of each beam are compared to determine if someone enters or exits through the door
Also detects if someone intended to pass but finally returned back
Respective entry/exit notifications are sent (over WiFi or LAN) to an aggregator application
CRF Feature formulation: based on enter/exit data notifications, the respective Dour Counting CRF Feature is calculated after normalization
Pressure mats:
Installed in pairs
operation mode similar with double-beam sensors above
Combining Double-beam + Pressure mats at the same Door
Pre-processing of information from both sensors at Arduino level
Expected to significantly increase the double-beam accuracy
Integration and validation of results on-going

6. Occupancy Extraction – Arduino-based sensors integration
Acoustic sensor:
Utilizing a low-cost analog acoustic sensor
Processing of 0,5sec samples, assigning a 0 or 1 value against a defined threshold
different per space (noisy/silent)
calculated after an initial calibration
20 consecutive samples form a 10-sec acoustic event sent to the aggregator application
CRF Feature formulation:
normalised from 0 - 1, counting the number of 1s
Either Raw (value at each time step) OR weighted average (e.g. last 3 timesteps)

7. Occupancy Extraction Test case Scenarios at CERTH
ITI Building 1st floor - Rest area:
Occupancy extraction utilizing:
one PIR (motion detector) and
one Acoustic sensor
Ground truth recorded for 4 days, by observing images captured by a webcam (manual occupancy annotation)
Can detect occupants:
Long-duration: having a short meeting at the tables OR
Short-duration: passing by / using the vending machine
Occupancy extracts presence/absense
Occupancy density information not extracted
12min meeting at the tables
People passing by
Test Scenario: Short meeting after 16:24

8. Occupancy Extraction Test case Scenarios at CERTH
Meeting Room:
Occupancy extraction utilizing:
one PIR (motion detector)
one Acoustic sensor
Double-Beam sensors at the door
Ground truth based on scheduling information and Observation recorded for 4 days
Occupancy extraction can detect exact number of occupants
Work in progress:
utilize input from CO2 sensor combined with a subset of above sensors (e.g. PIR + CO2)
validate produced accuracy on occupancy density, against extraction based on door counter sensors

9. Occupancy Extraction
Deployment of Kinect cameras in Developers office:
Tested various different Topologies
Kinect Limitations (Maximum Distance, Depth Image Limitations, etc.)
Overlapping FOV (Field Of View)
Office Limitations (wooden separators between the offices)
All entrances (2 Doors) must be in FOV
4 Kinect cameras
4 Kinect cameras
5 Kinect cameras

10. Occupancy Extraction
Deployment of Kinect cameras in Developers office:
4 Sub-Spaces created based on:
Thermal Zoning
Lighting Zoning
Offices’ Topology
Lowest Granularity Possible
Real-time occupancy extraction for 4 different sub-spaces, utilizing input from Kinect cameras only

11. Individual Occupant Extraction RFID
RFID Deployment in ITI/CERTH Pilot:
Initial Trial Tests started with demo equipment, using:
UHF Passive RFID 4-port Readers
UHF Passive external RFID Antennas
UHF RFID Tags:
Passive and
Semi-Passive (with Battery to strengthen return signal)

12. Individual Occupant Extraction RFID
2 Basic scenarios
One Antenna per Door <- This is the desired approach
Two Antennas per Door
higher accuracy, easier occupant’s direction extraction
BUT doubles cost
Individual Extraction Approach with RFID Tags based on:
Unique RFID ID
RSSI (Received Signal Strength Indicator)
Phase comparison (separation of moving or not moving individuals) (under investigation)

13. Individual Occupant Extraction RFID
Proposed RFID Topology, to cover ITI Building First Floor main selected pilot sites (utilizing one antenna per door)
8 x Antennas
2 x 4-port Reader

14. Individual Occupant Extraction RFID
Antenna
RFID Trial Deployments at different entrances of ITI Building First Floor
Tested different topologies for optimal direction extraction
Tested accuracy of RFID readings
both passive & semi-passive
Tested RFID Reader triggered operation:
Non-continuous reading
Activation from analog contact sensor (Double Beam/Pressure Mats at the door, PIR etc.)
Progress on-going
Pressure-Mat
for triggering RFID operation
RFID Trial Deployment
at ITI Building First Floor main entrance:
4-port Reader

15. Task Progress / Main Achievements Occupancy Prediction
Prototype development extending functionalities
Incorporation of scheduling information into occupancy prediction algorithm
New tests for the Markov and Semi-Markov approach with various parameterization based on
Occupancy Data from FP7 Adapt4EE Project
Occupancy Data from pilot sites (e.g. ITI Kitchen)
Evaluation of the prediction algorithm against various spatial combinations (e.g. considering only one space, dividing a space into sub-spaces etc.)

16. Occupancy Prediction - Schedule
Two-steps Occupancy Prediction calculation, incorporating scheduling information
Probabilistic
Occupancy Prediction
Rule-Based
Occupancy Prediction
Real-time Occupancy Input

17. Occupancy Prediction – Schedule Algorithm
Rule-based approach
Extract information for scheduled events from CERTH online reservation system database
Compare declared against actual values based on extracted occupancy data
Calculate various statistics (e.g. average start time delay/earliness, average end time difference)
Use statistics for prediction taking also into account real-time occupancy state
Initial validation of the approach based on reservations and occupancy data from CERTH Meeting Room

18. Occupancy Prediction – Schedule Example
Scheduled Event  11:00 – 12:00
1st scenario  Actual meeting duration: 10:50 – 11:55
a) Current time: Before meeting actual start time -> prediction based on statistics
Current time: Meeting started earlier than expected -> prediction is adjusted based on actual real time occupancy + statistics (for end time)

19. Occupancy Prediction – Schedule Example
Scheduled Event  11:00 – 12:00
2nd scenario  Actual meeting duration: 10:45 – 11:48 (ends very early)
a) Long time before meeting start time: prediction based on statistics
b) After meeting ends (observing occ. = 0 for some time) -> prediction catches the end of meeting

20. Occupancy Prediction – Schedule Example
Scheduled Event  11:00 – 12:00
3rd scenario  Meeting not held
a) Long time before meeting start time: prediction based on statistics
b) After some time of no occupancy detected -> prediction catches that the meeting will not be held.

21. Occupancy Prediction Evaluation - Adapt4EE data
Validation of implemented algorithms using collected Occupancy Data from FP7 Adapt4EE Project
Occupancy data used for model creation
from 2 Hospital Meeting rooms in Portugal
3-months period (February, March & April)
two days were left out for testing
Tested various scenarios including
expected events (happening on a regular basis)
unexpected events (not following typical average occupancy)
Models tested
Markov models using various parameterization
Semi-Markov models using various parameterization
Historical Average models

22. Occupancy Prediction Evaluation - Adapt4EE data
Historical Average Models
They seem to give sufficient results for long-term prediction (whole day or after some hours from current time).
They can predict only expected events that seem to happen on a regular basis.
Markov Model, Historical Average Model, Ground Truth
Whole day prediction

23. Occupancy Prediction Evaluation - Adapt4EE data
Markov Models
They seem to success in both expected and unexpected events prediction.
Parameterization has an important impact on the outcome  Optimal parameters values selection is needed for each space (during training period)
Markov Model, Average Model, Ground Truth
Regular event prediction

24. Occupancy Prediction Evaluation - Adapt4EE data
Markov Models
They perform better than Historical Average models for short-term predictions in most cases.
They are highly affected by current state.
The closest the current state is to an occupancy change the more accurate is the result.
Markov Model, Average Model, Ground Truth
Half-hour prediction

25. Occupancy Prediction Evaluation - Adapt4EE data
Semi-Markov Models
Generally they seem to perform worse than Markov Models.
They are relatively better for short-term unexpected events in comparison to other models (15min prediction time frame).
Semi-Markov Model, Ground Truth

26. Occupancy Prediction Evaluation - CERTH Pilot Kitchen Data
Created models using occupancy data for 3 months period (May, June & July) coming from Kitchen pilot area (3 days left out for testing).
Results similar to the ones based on Adapt4EE data.
Initial results indicate that both Markov and Average models can easily predict in high accuracy, areas with daily repeated occupancy patterns.
Markov, Average Model, Ground Truth
Whole day Kitchen Sink Prediction
Whole day Kitchen Tables Prediction

27. Occupancy Prediction Evaluation
Testing various spatial combinations:
Results are better when correlated spaces are examined together
Uncorrelated spaces can be examined separately (e.g. meeting room)
Planning/Pending Actions:
A hybrid model may be adopted using different types of models depending on the distance of prediction time from current time (e.g. after 2-3 hours apply average instead of Markov algorithm)
Validate based on more occupancy data collected from CERTH pilot sites / coming from various sensor combinations
Optimal selection of model parameters according to the uniqualities of each pilot area

28. Interfacing with IAM (TUK)
XSD schemas have been defined for all the services needed
Initial services have been implemented and tested on TUK server
Pending issues: implementation and testing of the rest services in cooperation with TUK (Peter - when?)

29. Interfacing with Linksmart (CNet)
Publishing events
Started publishing Occupancy_Change events from all DEMO CERTH areas (Kitchen, Developers Office, Meeting Room, Rest Area)
Started publishing Occupancy_Prediction_Change events from CERTH Kitchen – other remaining DEMO areas will be added within this week
Subscribing to events
Code for the subscription and receive of data from LS is ready
It should be incorporated in the corresponding code
A simple demonstration UI already available, presenting messages in a list box for testing/validation

30. Sample XML for Occupancy_Change Ebent

31. Sample XML for Occupancy_Prediction_Change

32. Deliverable D3.3
Deliverable D3.3 Ambient User Interfaces, User Behavioural Profiling and Activity Flow Framework
Combined with T3.5
Detailed structure already available
Initial version available with inputs from HYPERTECH on “UI and User Profiling sections”
CERTH will fully provide “Activity Flow Framework” section
Planning:
Peer review version available by 19/9

33. Planning for next efforts
Occupancy Extraction
Selection of CRF Features for CO2 sensor / finalise parameters for remaining sensors
Integrate CRF Featured from Kinect cameras occ .extraction
Further improve CRF performance (tests with extra features)
Initial accuracy validation per sensor combination
Implement Individual Extraction using RFID (based on CRF approach)
Occupancy Prediction
Finalize and integrate scheduling algorithm in overall Prediction Prototype (integrated with data from IAM)
Acquire more real-data from CERTH pilot premises (based on new installations) towards better assessment of the different approaches
Add individual prediction support (based on RFID extraction)

34. Planning for next efforts
Finalize integration with IAM + LinkSmart
Extend deployment of prototype continuously producing Occupancy Events from all predefined CERTH pilot-sites for the 2nd review
Preliminary comparison of different multi-sensor topologies, running tests in parallel with different data for the same areas

35. Questions & Discussion
Thank You !!!