BatTracker: High Precision Infrastructure-free Mobile Device Tracking in Indoor Environments Bing Zhou1, Mohammed Elbadry2, Ruipeng Gao3, Fan Ye1 1 ECE Department, Stony Brook University 2 CS Department, Stony Brook University 3 School of Software Engineering, Beijing Jiaotong University ACM SenSys 2017 Delft, The Netherlands Good morning everyone, I’m glad to be here to present our work on mobile device motion tracking, which is a joint work with my colleague Mohammed, my advisor prof. Fan Ye from Stony Brook University, and Ruipeng from Beijing Jiaotong university.

Motivation - Motion Tracking Video Gaming Virtual Reality Health Rehabilitation Tracking the continuous movements of a device in an indoor space is a form of human-computer interaction, popular in many applications including video gaming, health rehabilitation, and virtual reality. Using inertial sensors embedded in the device is the most straightforward approach, however, they suffer from large drift errors over time. To combat such errors, additional infrastructure is usually installed. For example, the state-of-the-art VR systems rely on multiple base stations or cameras to track the user in a small area. They increase the cost by several hundred dollars and require installation efforts.

Current Approaches Vision based Special hardware Lighting condition Computationally heavy Privacy issues Among the current tracking approaches, vision based methods are widely used in some products. However, they have certain limitations. For example, they usually require special hardware, such as the depth sensors. The performance is subject to lighting conditions and processing video is computationally heavy. Besides, cameras may raise privacy concerns. Microsoft XBOX360 Oculus VR

Current Approaches RF signals based Wi-Fi, RFID mmWave (e.g., 60GHz) Limited accuracy due to the high propagation speed mmWave (e.g., 60GHz) High accuracy while hardware is not available in most existing devices RF signals such as Wi-Fi or RFID are also used for tracking, but they have limited accuracy due to the high propagation speed. mmWave has high accuracy because of its high frequency, but the hardware is not available in most existing devices. Additionally, mmWave beam is highly directional and narrow, which makes it easy to be blocked. Image from Google Project Soli

Acoustic Approach Acoustic signal Low propagation speed High ranging accuracy Less privacy issue No image/video data captured Light computation Orders of magnitude less compared to vision method Existing hardware Almost all smart devices have speakers and microphones Acoustics is a favorable sensing modality for ranging and tracking due to the slow sound propagation speed, hence higher accuracy. It does not have the same privacy issue as cameras and it’s also light task for computation. What’s more, almost all the smart devices have speakers and microphones, which makes it possible for tracking without addition hardware.

BatTracker Design Speaker & Microphone Distance measurements Distance to reference objects In our paper, we propose BatTracker, the first high precision, infrastructure-free mobile device tracking system in indoor environments. Now, let’s consider the this typical VR tracking scenario. The system localizes the user by measuring the distances from the user to multiple base stations. What if we don’t have such stations? If we can remove these base stations, we can cut down the cost by a few hundred of dollars for a set of such system. As the name says, BatTracker localize the mobile device just like a Bat flying in the dark. At a high level, the device continuously emits acoustic signals that bounce off surfaces of nearby objects, such as walls, ceiling, and desks. The echoes are received and relative distances to those reference objects are inferred. And then we select a few reference objects which forms a coordinate, and keep tracking the distances variations to these reference object, thus localizing the device. Tracking these distances

Acoustic Sensing Time of arrival (Distance) Amplitude 1ms Emitting signal: Frequency 17KHz Duration 1ms Interval 30ms Hanning window 30ms Cross-correlate Time of arrival (Distance) 30ms Direct Path Amplitude Echo Received signal: Noise removed Echo Echo Frequency shift (Velocity) First, let’s look at how we obtain the accurate distance measurements from nearby objects. This figure shows the emitting signal design. We choose signal with a constant frequency of 17KHz, which is slightly audible to human. An even higher frequency decreases the robust tracking ranges because of faster signal power attenuation. By setting a moderate volume, our designed sound signal is nearly inaudible to most users. We choose a pulse length of 1ms. Such a short pulse length reduces potential overlapping between echoes travelling similar distances, thus improving the distance measurement resolution. To ensure echoes from two consecutive pulses do not overlap, there has to be enough gap in between, we leave 30ms. A Hanning window is applied on the pulse to reshape its envelop to increase its peak to side lobe ratio, thus producing higher signal to noise ratio. We apply short term fourier transform to get frequency shift, thus the velocity. STFT STFT STFT

Track Initiation Track Generation Track Association Final Selection 5 5 4 2 3 3 To start the tracking, we need to find three reference objects, which are large surfaces perpendicular to each other. Usually they can be two adjacent side walls and the ceiling in a room, or other larger furniture. This figure shows the distance measurements from acoustic sensing. The track initiation is the process of finding major reference objects with stable echo reflections, and determining the distances to each of them, such that we can track such distances continuously. 1 1 1 Correlate distance measurements with accelerometer data

Challenges Inertial sensors can help! Distance Measurements False track divergence Track crossing A. B. C. D. E. P2 P2 P1 P2 P4 P1 P3 P1 P1 P2 X P3 P3 P2 P1 P3 P5 P3 Y P4 P5 Due to the clutters in a room and moving dynamics, tracking the distance continuously over time is a challenging problem. For simplicity, let’s consider 2D tracking scenario. X and Y means the distance measurements to two objects, such as two walls perpendicular to each other. Inertial sensors can help! Missing data Tracks diverge after merge Time Naïve method 1: Continuous movement Naïve method 2: Continuous velocity Naïve method 3: Reliable measurements Assumptions:

Tracking Framework Overview Track Initiation Distance Measurements Current Tracks Multi-Hypothesis Tracking Motion Model Linear Acceleration Gyroscope Track Updating Observation Model Probabilistic Data Association Distance Candidates Track Splitting Amplitude Candidates Weighting and Resampling Doppler Shift Candidates Time Track Pruning Track Estimation

Multi-hypothesis Tracking Particle Filter Algorithm Track Pruning Track Splitting Initial Track Track Update Particle filter has been proven to be an effective algorithm framework for tracking problems. Each particle is a multi-dimensional vector representing one concrete estimation of the system state – in this case the position and velocity of the device. A traditional particle framework works like this. The have a motion model to update the location, which has a certain uncertainty. Then the device makes an observation, and based on this observation, it estimates the weight of each particle, and resample the particle sets. However, in our case, due to the clutter environment, we may get multiple distance measurements at each time stamp. Thus, we propose a particle filter based multi-hypothesis tracking algorithm for our case. Although MTH has been used in computer vision and radar applications, it is only a general framework, the critical task is how to adapt MHT to our specific inertial/acoustic tracking problem. Current state Landmark Predicted state from inertial data Validated measurements

Probabilistic Data Association Measurement likelihood: ω1 ω0 Track Update Incorporate velocity (Doppler shift) and amplitude: ω2 ω3 Echoes with similar distance usually have different velocity along different direction Amplitude tends to be continuous for echoes from same object L is the likelihood of measurement zm originating from the desired reference object rather than clutter, z hat is the predicted device position, S is the covariance matrix of the measurement. To enhance this probability estimation, we incorporate the velocity and amplitude data. We can predict the velocity from the motion model, and measure it from doppler shift. And for the amplitude, if it’s from the same direction, it should be continuous. We further estimate the data missing probability at each time slot according to the pose of the device. Thus, we normalize the weights, and assign them to each possible clouds. Data Missing Probability: PM(t) heavily related to the phone pose (holding gesture), We increase PM(t) when any two tracks are close to each other.

Evaluation - Setup We evaluate the system in a highly cluttered laboratory environment with a clean corner. A tilted box with drawn traces is used as the ground truth. We move the phone along the trace and evaluate the tracking accuracy. We also did similar experiments in a conference room with glass doors and a living room, and achieved similar results.

Evaluation - Ranging Accuracy 0.5m, 1m, 1.5m, 2m, 2.5m, 3m, 30 measurements at each location. ~1cm error at 90%, Maximum error is ~2cm. First, it’s the ranging accuracy. We hold the phone in front of a plain wall, and do the measurements at different distances, from 0.5m to 3m with a step length of 0.5m. And we did 30 measurements at each location. As the result shows, the error is with 1cm at 90%, and the maximum error is about 2cm. No major difference is observed with/without noise, which shows that our design is robust to ambient noise. Robust to ambient noise.

Evaluation - Inertial, Doppler and Ranging Now we evaluate some input measurements of the system, inertial sensors, doppler shifts, and direct ranging. We point the phone to a wall, and move it back and forth. The left figure shows the moving velocity derived from inertial sensor, doppler shift and acoustic ranging. We can see there’s a constant drift for the inertial data, and velocity from doppler shifts is reasonable but it has large variances as well. The right figure shows the device location. Inertial based tracking have large accumulated error.

Evaluation – 2D Tracking Accuracy For 2D tracking, we compare our approach to some recent acoustic tracking work. CAT uses a distributed FMCW approach to estimate the distance from the device to multiple external speakers, and triangulate the device location. While our approach automatically selects reference objects and measure the direct distances to them. AAMouse tracks the distances to multiple external speakers using doppler shifts, which has accumulated error. The experiment results show that our approach can achieve sb-cm accuracy, which is even higher than random ranging accuracy test. That’s because our algorithm helps remove outliers, and smooth the track. Sub-cm accuracy for 2D tracking, even higher than random ranging accuracy test! Smoothing nature of our algorithm helps remove outliers, and smooth the track. CAT triangulates the device position from distances to multiple speakers, which enlarges the error. AAMouse has accumulated error, while CAT and BatTracker do not have. CAT: Wenguang Mao, Jian He, and Lili Qiu. “CAT: high-precision acoustic motion tracking.” [MobiCom 2016] AAMouse: Sangki Yun, Yi-Chao Chen, and Lili Qiu. “Turning a mobile device into a mouse in the air.” [MobiSys 2015]

Evaluation – 3D Tracking Accuracy This figure shows the 3D tracking accuracy, we have ~1cm error at 90%.

Evaluation – Different Algorithms Tracking comparison: More drawing examples: This figure shows the comparison between different algorithms on the same data set. Figure (a) shows the tracking result if we choose the nearest neighbor in distance measurements at each time stamp, and the result quickly diverges. Figure (b) is the result based on inertial sensors only, which has a very large drift. Figure © is the result of a traditional particle filter with single hypothesis, it’s much better than the previous methods, but it diverges at the letter M. The last figure shows the result of our multi-hypothesis tracking approach, which shows the best result. These are more examples for the drawing, 2D writing, and 3D spiral drawing.

Evaluation – Efficiency Allocated memory and CPU usage on smartphone Tracking error and number of particles In terms of the computation efficiency, we implemented the core tracking algorithm on android, and test it on a Huawei P9, which is a middle end phone of 2016. As we can see, the algorithm only take ~12 MB memory.

Limitation Limited tracking range Device holding gesture Current design has a range of . Device holding gesture Quality omni-directional speaker/microphone may help. Reference Objects Require clean walls, large furniture such as closets, cabinets, and tables. Track loss problem As probabilistic algorithms are used, we still have chances for trace losing. We still have several limitations. First, our tracking range is limited due to the robust acoustic sensing range. In current design using smart phones, we need to pay careful attention to the holding gesture. Turning the phone into extreme angles will cause large data loss. Besides, we need large reference objects, which can be walls in a room, but they are not always available. Due to the above facts, we can still have track loss problem.

Future work Fast track recovery Utilize all the available objects Design a mechanism for automatic track loss detection and recovery. Utilize all the available objects Leverage all stable reflections Customized hardware Customized omnidirectional, high-sensitivity microphones Different devices More comprehensive tests on different smart phones Our future work focuses on improving tracking robustness, especially dealing with tack loss problem. For example, we can design some mechanism for fast track recovery, utilizing all the available objects, and leverage customized hardware for better performance. We also plan to do more tests on different smart devices.

Thank You.

Back Up Slides

Evaluation – Data Missing Probability Wall It’s also necessary to estimate the data missing probability under different device pose, especially the orientation. From our experiments, we found that data missing is not sensitive to the relative distance, but it’s very sensitive to the orientation of the device. The figure on the right side shows that if we rotate the phone along x or z axis, the data missing probability significantly changes. Thus, according to the experiment, we approximate the probability.

Evaluation – Impact of Initiation Error