WiFinger: Talk to Your Smart Devices with Finger-grained Gesture Hong Li, Wei Yang, Jiangxin Wang, Yang Xu, Liusheng Huang University of Science and Technology of China

Gesture Recognition SoundWave LeapMotion ArmTrack SoundWave: using the Doppler effect to sense gestures. I am a smartwatch and I can track my user’s arm. ArmTrack

WiFi signal is ubiquitous People can access to WiFi signals almost everywhere. WiFi can be effected by surrounding human activities.

Channel State Information(CSI) Receiver Sender Channel H x y 𝑦=𝐻×𝑥+𝑛

Channel State Information(CSI) CSI is an estimation of H Human Gesture and other movements can effect the propagation of multipath signals. H includes gesture information

What can we do with CSI? WiFall

What can we do with CSI? WiKey

Finger gesture recognition

Challenges How to detect and capture the subtle signal changes caused by micro- movements? How to extract distinguishable features? How to classify these features of different finger movements?

WiFinger Overview 动态时间规整 离散小波变换

CSI Capture Each packet corresponds to a CSI matrix H has a size of 30*1 CSI stream 𝐶=[ 𝐻 𝑡 1 , 𝐻 𝑡 2 ,⋯, 𝐻 𝑡 𝐿 ]

Preprocessing 带权重的滑动平均 Outlier Removal [𝜇−𝛾×𝜎, 𝜇+𝛾×𝜎] 𝜇是中值，𝜎是绝对中位差=𝑚𝑒𝑑𝑖𝑎𝑛( 𝑥 𝑖 −𝑚𝑒𝑑𝑖𝑎𝑛(𝑥)),𝛾是3 m=30，历史数据相关性 Outlier Removal [𝜇−𝛾×𝜎, 𝜇+𝛾×𝜎] Low-pass Filtering Cut-off frequency: 60Hz Weighted Moving Average

How to extract the gestures? Find a sign indicator of CSI stream Set a threshold of the sign indicator to detect the starting an finishing points

Gesture Extraction The sign indicator is defined as 𝔼 ℎ 2 2 / 𝛿 𝑞 2 Remove the DC component Cut the CSI stream with a sliding window Calculate the correlation matrix The sign indicator is defined as 𝔼 ℎ 2 2 / 𝛿 𝑞 2 ℎ 2 is the principal component, 𝑞 2 is the second eigenvector 𝛿 𝑞 2 = 1 𝑁 𝑐 −1 𝑙=2 𝑁 𝑐 | 𝑞 2 𝑙 − 𝑞 2 (𝑙−1)|

Threshold of sign indicator Guard interval 𝑇 𝑏 Smooth the amplitude with median filter Select the value of the third quartile of the sign indicator as the threshold

Feature Extraction Finger gesture profile P (30 * L) Combine P to Feature vector F (5 * L) F=[ 𝑓 1~6 , 𝑓 7~12 , 𝑓 13~18 , 𝑓 19~24 , 𝑓 25~30 ] 𝑓 𝑘~𝑙 = 1 𝑙−𝑘+1 𝑖=𝑘 𝑙 𝑝 𝑖 Compress F with DWT(Discrete Wavelet Transformation)

Discrete Wavelet Transformation 低通滤波器，高通滤波器，降采样滤波器

Discrete Wavelet Transformation Decompose the signal into a coarse approximation coefficients and detail coefficients 低通滤波器，高通滤波器，降采样滤波器 Compress the original signal while preserving both time and frequency domain information

Classification（kNN classifier） The distance is calculated by DTW(Dynamic Time Warping) 动态时间规整

Implementation & Evalutaion 1 directional TX antenna and 3 omni-directional RX antenna

Data Collection 10 users perform ASL gesture No.1 to No.9

Automatic Gesture Extraction Accuracy User 2 and user 7 separately have average gesture detect ratio of 76% and 81%. They cannot fully stretch their fingers when performing 3 and 9.

Gesture Recognition The average recognition accuracies of gesture 3 and 9 are relatively lower. The two gesture and hard to perform.

Gesture Recognition The recognition accuracy of user 5 is around 80%

Number Text Input Using WiFinger WiFinger achieves average recognition accuracy of 82.67%

CSI patterns for different gestures 两把椅子和一个柜子

Devices Positioning As the distance between transceivers increasing, the patterns nearly disappeared.

Frequency Band

Summary Design a scheme enables WiFi signals to realize continuously number text input by recognizing finger-grained gestures from ASL. Achieve high recognition accuracy with an average recognition accuracy 90.4% per user. It is a non-intrusive and device-free solution to finger-grained gestures recognition.