1. BatTracker: High Precision Infrastructure-free Mobile Device Tracking in Indoor Environments
Bing Zhou1, Mohammed Elbadry2, Ruipeng Gao3, Fan Ye1
1 ECE Department, Stony Brook University
2 CS Department, Stony Brook University
3 School of Software Engineering, Beijing Jiaotong University
ACM SenSys 2017
Delft, The Netherlands
Good morning everyone, I’m glad to be here to present our work on mobile device motion tracking, which is a joint work with my colleague Mohammed, my advisor prof. Fan Ye from Stony Brook University, and Ruipeng from Beijing Jiaotong university.

2. Motivation - Motion Tracking
Video Gaming
Virtual Reality
Health Rehabilitation
Tracking the continuous movements of a device in an indoor space is a form of human-computer interaction, popular in many applications including video gaming, health rehabilitation, and virtual reality. Using inertial sensors embedded in the device is the most straightforward approach, however, they suffer from large drift errors over time. To combat such errors, additional infrastructure is usually installed. For example, the state-of-the-art VR systems rely on multiple base stations or cameras to track the user in a small area. They increase the cost by several hundred dollars and require installation efforts.

3. Current Approaches Vision based Special hardware Lighting condition
Computationally heavy
Privacy issues
Among the current tracking approaches, vision based methods are widely used in some products. However, they have certain limitations. For example, they usually require special hardware, such as the depth sensors. The performance is subject to lighting conditions and processing video is computationally heavy. Besides, cameras may raise privacy concerns.
Microsoft XBOX360
Oculus VR

4. Current Approaches RF signals based Wi-Fi, RFID mmWave (e.g., 60GHz)
Limited accuracy due to the high propagation speed
mmWave (e.g., 60GHz)
High accuracy while hardware is not available in most existing devices
RF signals such as Wi-Fi or RFID are also used for tracking, but they have limited accuracy due to the high propagation speed. mmWave has high accuracy because of its high frequency, but the hardware is not available in most existing devices. Additionally, mmWave beam is highly directional and narrow, which makes it easy to be blocked.
Image from Google Project Soli

5. Acoustic Approach Acoustic signal Low propagation speed
High ranging accuracy
Less privacy issue
No image/video data captured
Light computation
Orders of magnitude less compared to vision method
Existing hardware
Almost all smart devices have speakers and microphones
Acoustics is a favorable sensing modality for ranging and tracking due to the slow sound propagation speed, hence higher accuracy. It does not have the same privacy issue as cameras and it’s also light task for computation. What’s more, almost all the smart devices have speakers and microphones, which makes it possible for tracking without addition hardware.

6. BatTracker Design Speaker & Microphone Distance measurements
Distance to reference objects
In our paper, we propose BatTracker, the first high precision, infrastructure-free mobile device tracking system in indoor environments. Now, let’s consider the this typical VR tracking scenario. The system localizes the user by measuring the distances from the user to multiple base stations. What if we don’t have such stations? If we can remove these base stations, we can cut down the cost by a few hundred of dollars for a set of such system.
As the name says, BatTracker localize the mobile device just like a Bat flying in the dark. At a high level, the device continuously emits acoustic signals that bounce off surfaces of nearby objects, such as walls, ceiling, and desks. The echoes are received and relative distances to those reference objects are inferred. And then we select a few reference objects which forms a coordinate, and keep tracking the distances variations to these reference object, thus localizing the device.
Tracking these distances

7. Acoustic Sensing Time of arrival (Distance) Amplitude
1ms
Emitting signal:
Frequency 17KHz
Duration 1ms
Interval 30ms
Hanning window
30ms
Cross-correlate
Time of arrival (Distance)
30ms
Direct Path
Amplitude
Echo
Received signal:
Noise removed
Echo
Echo
Frequency shift (Velocity)
First, let’s look at how we obtain the accurate distance measurements from nearby objects. This figure shows the emitting signal design.
We choose signal with a constant frequency of 17KHz, which is slightly audible to human. An even higher frequency decreases the robust tracking ranges because of faster signal power attenuation. By setting a moderate volume, our designed sound signal is nearly inaudible to most users.
We choose a pulse length of 1ms. Such a short pulse length reduces potential overlapping between echoes travelling similar distances, thus improving the distance measurement resolution.
To ensure echoes from two consecutive pulses do not overlap, there has to be enough gap in between, we leave 30ms. A Hanning window is applied on the pulse to reshape its envelop to increase its peak to side lobe ratio, thus producing higher signal to noise ratio.
We apply short term fourier transform to get frequency shift, thus the velocity.
STFT
STFT
STFT

8. Track Initiation
Track Generation
Track Association
Final Selection 5 5 4 2 3 3
To start the tracking, we need to find three reference objects, which are large surfaces perpendicular to each other. Usually they can be two adjacent side walls and the ceiling in a room, or other larger furniture. This figure shows the distance measurements from acoustic sensing. The track initiation is the process of finding major reference objects with stable echo reflections, and determining the distances to each of them, such that we can track such distances continuously. 1 1 1
Correlate distance measurements with accelerometer data

9. Challenges Inertial sensors can help! Distance Measurements
False track divergence
Track crossing A. B. C. D. E. P2 P2 P1 P2 P4 P1 P3 P1 P1 P2 X P3 P3 P2 P1 P3 P5 P3 Y P4 P5
Due to the clutters in a room and moving dynamics, tracking the distance continuously over time is a challenging problem. For simplicity, let’s consider 2D tracking scenario. X and Y means the distance measurements to two objects, such as two walls perpendicular to each other.
Inertial sensors can help!
Missing data
Tracks diverge after merge
Time
Naïve method 1:
Continuous movement
Naïve method 2:
Continuous velocity
Naïve method 3:
Reliable measurements
Assumptions:

10. Tracking Framework Overview
Track Initiation
Distance Measurements
Current Tracks
Multi-Hypothesis Tracking
Motion Model
Linear Acceleration
Gyroscope
Track Updating
Observation Model
Probabilistic Data Association
Distance Candidates
Track Splitting
Amplitude Candidates
Weighting and Resampling
Doppler Shift Candidates
Time
Track Pruning
Track Estimation

11. Multi-hypothesis Tracking
Particle Filter Algorithm
Track
Pruning
Track
Splitting
Initial
Track
Track
Update
Particle filter has been proven to be an effective algorithm framework for tracking problems. Each particle is a multi-dimensional vector representing one concrete estimation of the system state – in this case the position and velocity of the device. A traditional particle framework works like this. The have a motion model to update the location, which has a certain uncertainty. Then the device makes an observation, and based on this observation, it estimates the weight of each particle, and resample the particle sets.
However, in our case, due to the clutter environment, we may get multiple distance measurements at each time stamp. Thus, we propose a particle filter based multi-hypothesis tracking algorithm for our case. Although MTH has been used in computer vision and radar applications, it is only a general framework, the critical task is how to adapt MHT to our specific inertial/acoustic tracking problem.
Current state
Landmark
Predicted state from inertial data
Validated measurements

12. Probabilistic Data Association
Measurement likelihood: ω1 ω0
Track
Update
Incorporate velocity (Doppler shift) and amplitude: ω2 ω3
Echoes with similar distance usually have different velocity along different direction
Amplitude tends to be continuous for echoes from same object
L is the likelihood of measurement zm originating from the desired reference object rather than clutter, z hat is the predicted device position, S is the covariance matrix of the measurement. To enhance this probability estimation, we incorporate the velocity and amplitude data. We can predict the velocity from the motion model, and measure it from doppler shift. And for the amplitude, if it’s from the same direction, it should be continuous.
We further estimate the data missing probability at each time slot according to the pose of the device. Thus, we normalize the weights, and assign them to each possible clouds.
Data Missing Probability:
PM(t) heavily related to the phone pose (holding gesture),
We increase PM(t) when any two tracks are close to each other.

13. Evaluation - Setup
We evaluate the system in a highly cluttered laboratory environment with a clean corner. A tilted box with drawn traces is used as the ground truth. We move the phone along the trace and evaluate the tracking accuracy. We also did similar experiments in a conference room with glass doors and a living room, and achieved similar results.

14. Evaluation - Ranging Accuracy
0.5m, 1m, 1.5m, 2m, 2.5m, 3m,
30 measurements at each location.
~1cm error at 90%,
Maximum error is ~2cm.
First, it’s the ranging accuracy. We hold the phone in front of a plain wall, and do the measurements at different distances, from 0.5m to 3m with a step length of 0.5m. And we did 30 measurements at each location. As the result shows, the error is with 1cm at 90%, and the maximum error is about 2cm. No major difference is observed with/without noise, which shows that our design is robust to ambient noise.
Robust to ambient noise.

15. Evaluation - Inertial, Doppler and Ranging
Now we evaluate some input measurements of the system, inertial sensors, doppler shifts, and direct ranging. We point the phone to a wall, and move it back and forth. The left figure shows the moving velocity derived from inertial sensor, doppler shift and acoustic ranging. We can see there’s a constant drift for the inertial data, and velocity from doppler shifts is reasonable but it has large variances as well. The right figure shows the device location. Inertial based tracking have large accumulated error.

16. Evaluation – 2D Tracking Accuracy
For 2D tracking, we compare our approach to some recent acoustic tracking work. CAT uses a distributed FMCW approach to estimate the distance from the device to multiple external speakers, and triangulate the device location. While our approach automatically selects reference objects and measure the direct distances to them. AAMouse tracks the distances to multiple external speakers using doppler shifts, which has accumulated error. The experiment results show that our approach can achieve sb-cm accuracy, which is even higher than random ranging accuracy test. That’s because our algorithm helps remove outliers, and smooth the track.
Sub-cm accuracy for 2D tracking, even higher than random ranging accuracy test! Smoothing nature of our algorithm helps remove outliers, and smooth the track.
CAT triangulates the device position from distances to multiple speakers, which enlarges the error.
AAMouse has accumulated error, while CAT and BatTracker do not have.
CAT: Wenguang Mao, Jian He, and Lili Qiu. “CAT: high-precision acoustic motion tracking.” [MobiCom 2016]
AAMouse: Sangki Yun, Yi-Chao Chen, and Lili Qiu. “Turning a mobile device into a mouse in the air.” [MobiSys 2015]

17. Evaluation – 3D Tracking Accuracy
This figure shows the 3D tracking accuracy, we have ~1cm error at 90%.

18. Evaluation – Different Algorithms
Tracking comparison:
More drawing examples:
This figure shows the comparison between different algorithms on the same data set. Figure (a) shows the tracking result if we choose the nearest neighbor in distance measurements at each time stamp, and the result quickly diverges. Figure (b) is the result based on inertial sensors only, which has a very large drift. Figure © is the result of a traditional particle filter with single hypothesis, it’s much better than the previous methods, but it diverges at the letter M. The last figure shows the result of our multi-hypothesis tracking approach, which shows the best result.
These are more examples for the drawing, 2D writing, and 3D spiral drawing.

19. Evaluation – Efficiency
Allocated memory and CPU usage on smartphone
Tracking error and number of particles
In terms of the computation efficiency, we implemented the core tracking algorithm on android, and test it on a Huawei P9, which is a middle end phone of As we can see, the algorithm only take ~12 MB memory.

20. Limitation Limited tracking range Device holding gesture
Current design has a range of
Device holding gesture
Quality omni-directional speaker/microphone may help.
Reference Objects
Require clean walls, large furniture such as closets, cabinets, and tables.
Track loss problem
As probabilistic algorithms are used, we still have chances for trace losing.
We still have several limitations. First, our tracking range is limited due to the robust acoustic sensing range. In current design using smart phones, we need to pay careful attention to the holding gesture. Turning the phone into extreme angles will cause large data loss. Besides, we need large reference objects, which can be walls in a room, but they are not always available. Due to the above facts, we can still have track loss problem.

21. Future work Fast track recovery Utilize all the available objects
Design a mechanism for automatic track loss detection and recovery.
Utilize all the available objects
Leverage all stable reflections
Customized hardware
Customized omnidirectional, high-sensitivity microphones
Different devices
More comprehensive tests on different smart phones
Our future work focuses on improving tracking robustness, especially dealing with tack loss problem. For example, we can design some mechanism for fast track recovery, utilizing all the available objects, and leverage customized hardware for better performance. We also plan to do more tests on different smart devices.

22. Thank You.

23. Back Up Slides

24. Evaluation – Data Missing Probability
Wall
It’s also necessary to estimate the data missing probability under different device pose, especially the orientation. From our experiments, we found that data missing is not sensitive to the relative distance, but it’s very sensitive to the orientation of the device. The figure on the right side shows that if we rotate the phone along x or z axis, the data missing probability significantly changes. Thus, according to the experiment, we approximate the probability.

25. Evaluation – Impact of Initiation Error