Whole-Home Gesture Recognition Using Wireless Signals —— MobiCom’13 Author: Qifan Pu et al. University of Washington Presenter: Yanyuan Qin & Zhitong Fei

Why interactive technique?  Control music volume with gesture when showering or cooking.  Turn on the thermostat with gesture while in bed.  The disabled need it more. Options?  Kinect at each room ? Expensive…  Wearable Devices? Wear a device in a shower?… Background

Can we use ？ Can we use it to recognize gestures?

How to detect gestures using Wi-Fi? Wi-Fi Doppler shifts Humans reflect Wi-Fi signals, thus can be treated as signal sources Human motion introduce Wifi Doppler shifts Different gestures exhibit different patterns.

Two Challenges Challenge 1: How to detect small Doppler shifts within WiFi’s bandwidth (typically 20MHz) Challenge 2: How to determine the relationship between Doppler shifts and gestures In WiSee the target human performs a repetitive gesture, which we use as that person’s preamble.

WiSee WiSee is a wireless system that enable whole-home gesture reconigtion. Three main questions: First, how does WiSee extract Doppler shifts from conventional wireless signals like Wi-Fi? Second, how does it map the Doppler shifts to the gestures performed by the user? Third, how does it enable gesture recognition in the presence of other humans in the environment?

Extracting Doppler shifts from Wireless Signals Doppler shift is the change in the observed frequency as the transmitter and the receiver move relative to each other. Move towards crests arrive at a faster rate Move from crests arrive at a slower rate Where c is the speed of light in the medium and f is the transmitter’s center frequency.

Extracting Doppler shifts from Wireless Signals How does WiSee deal with frequency offsets?

Mapping Doppler Shifts to Gestures WiSee extracts the Doppler information by computing the frequency- time Doppler profile of the narrowband signal.

Segmentation Gestures WiSee leverages the stucture of the Doppler profiles. Set of segments that have positive and negative Doppler shifts.

Gestures Classification The receiver can classify gestures by matching the pattern. Positive “1” Negative “-1” Both “2”

Challenge 3: Interference from other people in the environment WiSee uses MIMO in n to improve accuracy But MIMO requires a known preamble In WiSee, a repeated gesture acts as preamble to specify certain user User pushes hand toward and away WiSee detect target user WiSee iteratively finds MIMO channel that maximizes Doppler energy

Multi-path Effects The reflections usually arrive at the receiver along multiple paths. A gesture towards receiver can create both positive and negative Doppler shifts at the receiver. WiSee automatically finds a proper MIMO direction because of iteration algorithm Strong reflectors (metal) can flip the positive and negative Doppler shifts. The receiver can calibrate the sign of the subsequent Doppler shifts based on preamble gestures.

Experiments Scenarios  Office building  Two-bed apt Many conditions  Line-of-sight, non-line-of-sight, through-the-wall, through-the-corridor, through-the-room

TX: transmitter RX: Wisee receiver Wall Feasibility of Wireless Gesture Detection

Gesture Recognition in Two-bed apt

Gesture Recognition Accuracy Confusion matrix

Gestures in the Presence of Other Humans (No target user, only other 12 people) False detection rate for a whole day

Influence from the number of interfering user (Target user and other people)

Influence from the distance of interfering user

Summary Design and evaluate, WiSee, a gesture recognition system that leverages wireless signals to enable whole-home sensing and recognition of human gestures WiSee can extract a rich set of gesture information from wireless signals and enable whole-home gesture recognition using only two wireless sources placed in the living room

Disadvantages  Whole-home gesture recognition with few wireless sources.  Without sensing device on human body or many devices in environment  High accuracy: 94%(on average) for 9 gestures. Advantages  Heavily influenced by the number of other people.  The number of gestures are limited (9 gestures) and hard to extend.  Using scale is smaller than wearable devices. Comments on WiSee

Future Works Security? Information fusion: Add other information to make the system more robust, like sound Explore whether the shape of body will affect the performance.