1. ❖ PEM-ID: Identifying People by Gait-Matching using Cameras and Wearable Accelerometers Thiago Teixeira, Deokwoo Jung, Gershon Dublon, Andreas Savvides Yale ENALAB

2. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab2 Introduction  Can we uniquely identify people in camera networks? (in cooperative enviroments)  Motivation:  Assisted Living identify people in a home  Security locate personnel  Corporate environments track facility usage  Plus, obtaining data traces for research:  Yale BehaviorScope project

3. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab3 Main Idea  Equip each person of interest with a wearable accelerometer node (with known ID)  Extract “motion signature” from:  each accelerometer  unique ID  each track  Position  Find pairs of matching signatures to obtain ID+Position

4. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab4 Problem Statement  Given: a set {S A i } of accelerometer signals and a set {S C j } of tracks extracted from a camera network  Find: the match matrix Λ which globally maximizes the similarity between pairs of signals S A i and S C j  Main assumptions:  Tracker: provides correct tracks in segments ≳ 4 steps  Camera placement: oblique from top (typical CCTV)  Occlusions: short-lived 01 10 00 Λ = tracks accelerometers

5. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab5 Challenge: motion signature  Motion paths can be subdivided into two types:  Transition motion  Starting, stopping, turning, changing speed  Large changes in tangential and centripetal acceleration  Cruising motion  Approximately same-speed linear motion  Only small-scale changes in acceleration  Gait  Comprises majority of time  Intuition: to ID people most of the time, use gait  Challenge: Nodes are not time-synchronized, have limited processors and low bandwith

6. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab6 Correlating Gait Signals from Asynchronous Sources  Sample-oriented methods are unsuitable for WSNs: (eg. Pearson's corr. coefficient, mutual information)  Fail given time synchronization offsets (or must slide one of the signals and recalculate)  Require a large number of samples to converge  Requires resampling/interpolation if signals have different sampling frequencies and/or phases  We can do better, using gait frequency and phase…

7. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab7 Timestamps of Gait Landmarks  Idea: Compare timestamps of heel-strike and midswing moments of gait:  H = (t H 0, t H 1, … )  M = (t M 0, t M 1, … )  From accels., and cameras:  S A i = {H A i, M A i }  S C j = {H C j, M C j }  Next step: define time-noise independent metric (offset and jitter)

8. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab8 Distance metric  Define distance from timestamp to sequence:  Then from sequence to sequence:  Then two metrics describing time offset and jitter:

9. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab9 Global Optimization  Invariance to time offset, timestamp noise  Global Optimization

10. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab10 Multiple-Person Simulations  We recorded 24 one-person traces:  12 × walking straight in different directions  12 × walking and turning in different directions  We overlapped multiple single-person traces with random time offsets (up to 1s) to simulate multiple- person scenarios:

11. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab11 Three-Person Experiments  Three people walking through FOV  One person wearing an accelerometer  Average recognition rate: 87.5% http://enaweb.eng.yale.edu/drupal/PEM-ID-videos

12. Thiago TeixeiraYale ENALAB - http://www.eng.yale.edu/enalab12 Conclusion  Presented a method to ID people in videos using accelerometers  Accuracy > 83%, for up to 10 people + 10 accels  Currently adapting for indoor use  Much smaller FOV  multiple cameras  Occlusions  use additional features

13. ❖ Thank you. Questions? BehaviorScope: http://www.eng.yale.edu/enalab/behaviorscope.htm Videos: http://enaweb.eng.yale.edu/drupal/PEM-ID-videos