WiFi Action Recognition via Vision-based Methods Jen-Yin Chang, Kuan-Ying Lee, Kate Ching-Ju Lin*, Winston Hsu Communication and Multimedia Lab National.

Slides:



Advertisements
Similar presentations
Distinctive Image Features from Scale-Invariant Keypoints
Advertisements

Feature Detection. Description Localization More Points Robust to occlusion Works with less texture More Repeatable Robust detection Precise localization.
Distinctive Image Features from Scale-Invariant Keypoints David Lowe.
Presented by Xinyu Chang
Whole-Home Gesture Recognition Using Wireless Signals —— MobiCom’13 Author: Qifan Pu et al. University of Washington Presenter: Yanyuan Qin & Zhitong Fei.
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
Computer vision: models, learning and inference Chapter 13 Image preprocessing and feature extraction.
Outline SIFT Background SIFT Extraction Application in Content Based Image Search Conclusion.
Object Recognition using Invariant Local Features Applications l Mobile robots, driver assistance l Cell phone location or object recognition l Panoramas,
Distinctive Image Features from Scale- Invariant Keypoints Mohammad-Amin Ahantab Technische Universität München, Germany.
SURF: Speeded-Up Robust Features
A Study of Approaches for Object Recognition
Object Recognition with Invariant Features n Definition: Identify objects or scenes and determine their pose and model parameters n Applications l Industrial.
Detecting Image Region Duplication Using SIFT Features March 16, ICASSP 2010 Dallas, TX Xunyu Pan and Siwei Lyu Computer Science Department University.
Distinctive Image Feature from Scale-Invariant KeyPoints
Object Recognition Using Distinctive Image Feature From Scale-Invariant Key point D. Lowe, IJCV 2004 Presenting – Anat Kaspi.
Scale Invariant Feature Transform (SIFT)
Blob detection.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Multiple Object Class Detection with a Generative Model K. Mikolajczyk, B. Leibe and B. Schiele Carolina Galleguillos.
Distinctive Image Features from Scale-Invariant Keypoints David G. Lowe – IJCV 2004 Brien Flewelling CPSC 643 Presentation 1.
Scale-Invariant Feature Transform (SIFT) Jinxiang Chai.
Multiclass object recognition
Bag of Video-Words Video Representation
Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.
Computer vision.
Internet-scale Imagery for Graphics and Vision James Hays cs195g Computational Photography Brown University, Spring 2010.
Object Tracking/Recognition using Invariant Local Features Applications l Mobile robots, driver assistance l Cell phone location or object recognition.
Reporter: Fei-Fei Chen. Wide-baseline matching Object recognition Texture recognition Scene classification Robot wandering Motion tracking.
Classifying Images with Visual/Textual Cues By Steven Kappes and Yan Cao.
Pedestrian Detection and Localization
CVPR 2003 Tutorial Recognition and Matching Based on Local Invariant Features David Lowe Computer Science Department University of British Columbia.
CSCE 643 Computer Vision: Extractions of Image Features Jinxiang Chai.
1 Research Question  Can a vision-based mobile robot  with limited computation and memory,  and rapidly varying camera positions,  operate autonomously.
Kylie Gorman WEEK 1-2 REVIEW. CONVERTING AN IMAGE FROM RGB TO HSV AND DISPLAY CHANNELS.
Distinctive Image Features from Scale-Invariant Keypoints David Lowe Presented by Tony X. Han March 11, 2008.
Jack Pinches INFO410 & INFO350 S INFORMATION SCIENCE Computer Vision I.
Automated Fingertip Detection
A Tutorial on using SIFT Presented by Jimmy Huff (Slightly modified by Josiah Yoder for Winter )
Scale Invariant Feature Transform (SIFT)
A Statistical Approach to Texture Classification Nicholas Chan Heather Dunlop Project Dec. 14, 2005.
Goggle Gist on the Google Phone A Content-based image retrieval system for the Google phone Manu Viswanathan Chin-Kai Chang Ji Hyun Moon.
Finding Clusters within a Class to Improve Classification Accuracy Literature Survey Yong Jae Lee 3/6/08.
Distinctive Image Features from Scale-Invariant Keypoints Presenter :JIA-HONG,DONG Advisor : Yen- Ting, Chen 1 David G. Lowe International Journal of Computer.
Blob detection.
SIFT.
Face recognition using Histograms of Oriented Gradients
SIFT Scale-Invariant Feature Transform David Lowe
CS262: Computer Vision Lect 09: SIFT Descriptors
Interest Points EE/CSE 576 Linda Shapiro.
Presented by David Lee 3/20/2006
Lecture 07 13/12/2011 Shai Avidan הבהרה: החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
Distinctive Image Features from Scale-Invariant Keypoints
Bag-of-Visual-Words Based Feature Extraction
Scale Invariant Feature Transform (SIFT)
Data Driven Attributes for Action Detection
SIFT on GPU Changchang Wu 5/8/2007.
Cheng-Ming Huang, Wen-Hung Liao Department of Computer Science
Assistive System Progress Report 1
CAP 5415 Computer Vision Fall 2012 Dr. Mubarak Shah Lecture-5
From a presentation by Jimmy Huff Modified by Josiah Yoder
Aim of the project Take your image Submit it to the search engine
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
SIFT keypoint detection
SIFT.
Keystroke Recognition using Wi-Fi Signals
SIFT SIFT is an carefully designed procedure with empirically determined parameters for the invariant and distinctive features.
Presented by Xu Miao April 20, 2005
Presentation transcript:

WiFi Action Recognition via Vision-based Methods Jen-Yin Chang, Kuan-Ying Lee, Kate Ching-Ju Lin*, Winston Hsu Communication and Multimedia Lab National Taiwan University, Taipei, Taiwan Academia Sinica, Taipei, Taiwan* 1 “WIFI ACTION RECOGNITION VIA VISION-BASED METHODS” - ICASSP2016

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion Outline 2

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion Motivation Preliminaries Framework Experiments Conclusion 3

Action Recognition 4 Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion Different Approaches Running 5 Reference: Images from Google Search

Limitation of Cameras Motivation Preliminaries Framework Experiments Conclusion ● Privacy Issue ● Line of Sight ● Darkness 6 Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion Limitation of Sensor Data ● Inconvenience ● shower, etc. 7 Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion Advantages of WiFi Signals 8

Motivation Preliminaries Framework Experiments Conclusion Advantages of WiFi Signals 9 Reference: Images from Google Search

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion 10 Motivation Preliminaries Framework Experiments Conclusion

Scenario Transmitter Reflected signals Signal Analysis 11 Receiver Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion Channel State Information (CSI) 12 Transmitter Receiver Reference: Images from Google Search

13 Time Subcarrier CSI Visualization Motivation Preliminaries Framework Experiments Conclusion

CSI Visualization Stand still ClapBox 14

Motivation Preliminaries Framework Experiments Conclusion Gabor Filters Different orientations Different scales 15 Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion Gabor Filters 16 mean + std 16 Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion BoW-SIFT 128-dimension vector SIFT FeatureK-meansHistogram 17 Reference: Images from Google Search

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion Motivation Preliminaries Framework Experiments Conclusion 18

Motivation Preliminaries Framework Experiments Conclusion 19

Motivation Preliminaries Framework Experiments Conclusion CSI Collection ● Tx: MacBook Pro 2014 ● Rx: Fujitsu SH560 (with Intel 5300 NIC) ● CSI matrix dimension ● (N tx x N rx x N subcarrier ) x N sample = (2 x 2 x 30) x t 20

Motivation Preliminaries Framework Experiments Conclusion Pre-processing ● Interpolation ● unevenly distributed packets ● Low-pass filtering ● remove high-frequency noises ● Window-averaging ● power distributions vary 21

Motivation Preliminaries Framework Experiments Conclusion Feature Extraction ● Gabor filter bank ● filter size = 15 ● feature dimension = N orientation x N scale x {mean, std} = 6 x 8 x 2 = 96 ● BoW-SIFT ● 48 centroid ● feature dimension = 48 22

Motivation Preliminaries Framework Experiments Conclusion SVM Learning – Feature revisited 23 Transmitter Receiver

Motivation Preliminaries Framework Experiments Conclusion SVM Learning – Early Fusion = Reference: Images from Google Search

Motivation Preliminaries Framework Experiments Conclusion SVM Learning – Late Fusion 25 Reference: Images from Google Search

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion Motivation Preliminaries Framework Experiments Conclusion 26

Motivation Preliminaries Framework Experiments Conclusion Datasets ● Dataset A ● 7 actions (Box, Clap, Wave hand, Kick, Quick squat, Jump, Stand still) ● 6 locations ● Dataset B ● to compare with KTH ● 7 actions (Box, Clap Wave hand, Walk, Jog, Run, Stand still) ● 2 persons 27 Reference: KTH – Christian Schuldt et al, “Recognizing human actions: A local svm approach”

Motivation Preliminaries Framework Experiments Conclusion Setting 28

Motivation Preliminaries Framework Experiments Conclusion Gabor is better than BoW-SIFT 29

Motivation Preliminaries Framework Experiments Conclusion Location dependency 30

Motivation Preliminaries Framework Experiments Conclusion Location dependency 31

Motivation Preliminaries Framework Experiments Conclusion Late fusion outperforms early fusion 32

● Motivation ● Preliminaries ● Framework ● Experiments ● Conclusion Motivation Preliminaries Framework Experiments Conclusion 33

Motivation Preliminaries Framework Experiments Conclusion Bridging WiFi and Vision ● The proposed vision-based method achieves accuracy above 85% identifying 7 actions, showing the promising performance of learning-based method ● Instead of traditional signal processing, we further aim to discover more general attributes from Wi-Fi signals by DNN-based methods 34

Thanks! Questions? 35 Eric: Ryan: “WIFI ACTION RECOGNITION VIA VISION-BASED METHODS” - ICASSP2016

Appendix 36

Wi-Fi Action Recognition Principle ● Signal properties vary through propagation ● power decay ● frequency drift ● phase shift ● Environment alter paths ● absorption, scatter and reflection ● Action changes environment ● different changes with distinct actions 37

● Performance retains when size decreases ● Performance drops when sizes unmatched Motivation Preliminaries Framework Experiments Conclusion Results on Resolution 38

Doppler Effect Channel State Information Amplitude Motivation Preliminaries Framework Experiments Conclusion 39

Qifan Pu et al., 2013 relative velocity of object and source require costly computation Channel State Information Amplitude Motivation Preliminaries Framework Experiments Conclusion 40

Qifan Pu et al., 2013 relative velocity of object and source require costly computation Channel State Information Rajalakshmi Nandakumar et al., 2014 changes of signal amplitude only coarse-grained actions Motivation Preliminaries Framework Experiments Conclusion 41

Qifan Pu et al., 2013 relative velocity of object and source require costly computation Wei Wang et al., 2015 more fine-grained information Rajalakshmi Nandakumar et al., 2014 changes of signal amplitude only coarse-grained actions Motivation Preliminaries Framework Experiments Conclusion 42

●Yan Wang et al., 2014 applied CSI distribution/sequence as features to recognize activity/walking trajectory labeled by users Motivation Preliminaries Framework Experiments Conclusion Previous work on CSI 43

● Wei Wang et al., 2015 proposed a PCA-based denoise method followed by DWT for hand-crafted features Motivation Preliminaries Framework Experiments Conclusion Previous work on CSI (cont.) 44

Motivation Preliminaries Framework Experiments Conclusion CSI and Texture ● Faster speed => larger phase change => denser stripes ● Actions of different speeds => different textures 45

Motivation Preliminaries Framework Experiments Conclusion Gabor Filters ●A 2D Gabor filter is a sinusoidal plane wave multiplied by a Gaussian kernel ●filters can closely mimic human visual system ●particularly appropriate for texture discrimination 46

Motivation Preliminaries Framework Experiments Conclusion SIFT-BoF ●SIFT ●detect and describe key points in an image ●each key point as 128-d feature ●bow-sift ●use K-means to cluster 128-d features to K centroids ●assign each key point to the nearest centroid ●for each image, build a K-dimensional histogram as feature 47

Scale Invariant Feature Transform (SIFT) ● Local feature = Detector + Descriptor ● Detector ● Find extremas ● Locate keypoints filtering extremas ● Descriptor ● Orientation Assignment ● remove rotation effect ● Create descriptor ● histogram of orientation 48

SIFT – Difference of Gaussian 49 Reference: Images from Google Search

SIFT – Extrema Finding 50 Reference: Images from Google Search

SIFT – Descriptor ● Compute gradient orientation histograms of sub- regions 51 Reference: Images from Google Search

Rooms 52

Motivation Preliminaries Framework Experiments Conclusion Image Transformation ● Convert CSI into image following Jet colormap 53