Online sketches with random features

Slides:

Advertisements

Similar presentations

Scaling Multivariate Statistics to Massive Data Algorithmic problems and approaches Alexander Gray Georgia Institute of Technology

Advertisements

Change-Point Detection Techniques for Piecewise Locally Stationary Time Series Michael Last National Institute of Statistical Sciences Talk for Midyear.

Feature Selection as Relevant Information Encoding Naftali Tishby School of Computer Science and Engineering The Hebrew University, Jerusalem, Israel NIPS.

Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.

Classification using intersection kernel SVMs is efficient Joint work with Subhransu Maji and Alex Berg Jitendra Malik UC Berkeley.

Mining Distance-Based Outliers in Near Linear Time with Randomization and a Simple Pruning Rule Stephen D. Bay 1 and Mark Schwabacher 2 1 Institute for.

An Overview of Machine Learning

Pitch Prediction From MFCC Vectors for Speech Reconstruction Xu shao and Ben Milner School of Computing Sciences, University of East Anglia, UK Presented.

Learning Inhomogeneous Gibbs Models Ce Liu

Low Complexity Keypoint Recognition and Pose Estimation Vincent Lepetit.

2008 SIAM Conference on Imaging Science July 7, 2008 Jason A. Palmer

GMM-Based Multimodal Biometric Verification Yannis Stylianou Yannis Pantazis Felipe Calderero Pedro Larroy François Severin Sascha Schimke Rolando Bonal.

Fast intersection kernel SVMs for Realtime Object Detection

Introduction to Sampling based inference and MCMC Ata Kaban School of Computer Science The University of Birmingham.

Feature Vector Selection and Use With Hidden Markov Models to Identify Frequency-Modulated Bioacoustic Signals Amidst Noise T. Scott Brandes IEEE Transactions.

6/26/2006CGI'06, Hangzhou China1 Sub-sampling for Efficient Spectral Mesh Processing Rong Liu, Varun Jain and Hao Zhang GrUVi lab, Simon Fraser University,

Fast High-Dimensional Feature Matching for Object Recognition David Lowe Computer Science Department University of British Columbia.

Modeling Pixel Process with Scale Invariant Local Patterns for Background Subtraction in Complex Scenes (CVPR’10) Shengcai Liao, Guoying Zhao, Vili Kellokumpu,

Simulation Where real stuff starts. ToC 1.What, transience, stationarity 2.How, discrete event, recurrence 3.Accuracy of output 4.Monte Carlo 5.Random.

Lecture 20 Object recognition I

Lecture 5: Learning models using EM

Detecting Image Region Duplication Using SIFT Features March 16, ICASSP 2010 Dallas, TX Xunyu Pan and Siwei Lyu Computer Science Department University.

Lesson #17 Sampling Distributions. The mean of a sampling distribution is called the expected value of the statistic. The standard deviation of a sampling.

Student: Hsu-Yung Cheng Advisor: Jenq-Neng Hwang, Professor

Scalable Training of Mixture Models via Coresets Daniel Feldman Matthew Faulkner Andreas Krause MIT.

Collaborative Ordinal Regression Shipeng Yu Joint work with Kai Yu, Volker Tresp and Hans-Peter Kriegel University of Munich, Germany Siemens Corporate.

Optimal Adaptation for Statistical Classifiers Xiao Li.

Spatio-chromatic image content descriptors and their analysis using Extreme Value theory Vasileios Zografos and Reiner Lenz

Heather Dunlop : Advanced Perception January 25, 2006

Real Time Abnormal Motion Detection in Surveillance Video Nahum Kiryati Tammy Riklin Raviv Yan Ivanchenko Shay Rochel Vision and Image Analysis Laboratory.

Image Processing David Kauchak cs458 Fall 2012 Empirical Evaluation of Dissimilarity Measures for Color and Texture Jan Puzicha, Joachim M. Buhmann, Yossi.

All features considered separately are relevant in a speech / music classification task. The fusion allows to raise the accuracy rate up to 94% for speech.

Deep Learning for Speech and Language Yoshua Bengio, U. Montreal NIPS’2009 Workshop on Deep Learning for Speech Recognition and Related Applications December.

Selective Block Minimization for Faster Convergence of Limited Memory Large-scale Linear Models Kai-Wei Chang and Dan Roth Experiment Settings Block Minimization.

Memory Bounded Inference on Topic Models Paper by R. Gomes, M. Welling, and P. Perona Included in Proceedings of ICML 2008 Presentation by Eric Wang 1/9/2009.

Machine Learning Using Support Vector Machines (Paper Review) Presented to: Prof. Dr. Mohamed Batouche Prepared By: Asma B. Al-Saleh Amani A. Al-Ajlan.

MUMT611: Music Information Acquisition, Preservation, and Retrieval Presentation on Timbre Similarity Alexandre Savard March 2006.

Jun-Won Suh Intelligent Electronic Systems Human and Systems Engineering Department of Electrical and Computer Engineering Speaker Verification System.

Compressible priors for high-dimensional statistics Volkan Cevher LIONS/Laboratory for Information and Inference Systems

Visual Categorization With Bags of Keypoints Original Authors: G. Csurka, C.R. Dance, L. Fan, J. Willamowski, C. Bray ECCV Workshop on Statistical Learning.

Epitomic Location Recognition A generative approach for location recognition K. Ni, A. Kannan, A. Criminisi and J. Winn In proc. CVPR Anchorage,

Powerpoint Templates Page 1 Powerpoint Templates Scalable Text Classification with Sparse Generative Modeling Antti PuurulaWaikato University.

Full-rank Gaussian modeling of convolutive audio mixtures applied to source separation Ngoc Q. K. Duong, Supervisor: R. Gribonval and E. Vincent METISS.

A multiline LTE inversion using PCA Marian Martínez González.

Speaker Verification Using Adapted GMM Presented by CWJ 2000/8/16.

Martina Uray Heinz Mayer Joanneum Research Graz Institute of Digital Image Processing Horst Bischof Graz University of Technology Institute for Computer.

ベーテ自由エネルギーに対するCCCPアルゴリズムの拡張

Dense-Region Based Compact Data Cube

SketchVisor: Robust Network Measurement for Software Packet Processing

Introduction to Sampling based inference and MCMC

Intrinsic Data Geometry from a Training Set

Online Multiscale Dynamic Topic Models

Visual exploratory data analysis:

A Forest of Sensors: Using adaptive tracking to classify and monitor activities in a site Eric Grimson AI Lab, Massachusetts Institute of Technology

Supervised Time Series Pattern Discovery through Local Importance

Non-Parametric Models

Visual exploratory data analysis:

Training Techniques for Deep Neural Networks

Cheng-Ming Huang, Wen-Hung Liao Department of Computer Science

PixelGAN Autoencoders

Presented by: Yang Yu Spatiotemporal GMM for Background Subtraction with Superpixel Hierarchy Mingliang Chen, Xing Wei, Qingxiong.

Filtering and State Estimation: Basic Concepts

PRAKASH CHOCKALINGAM, NALIN PRADEEP, AND STAN BIRCHFIELD

Feature Selection Analysis

Decision Making Based on Cohort Scores for

Missing feature theory

Sketched Learning from Random Features Moments

N. Keriven1, D. Garreau2, I. Poli3

Recognizing Deformable Shapes

Gentle Measurement of Quantum States and Differential Privacy *

Presentation transcript:

Online sketches with random features Nicolas Keriven Ecole Normale Supérieure (Paris) CFM-ENS chair in Data Science, « Laplace » post-doc (PhD with Rémi Gribonval at Inria Rennes) Paris Science and Data, Apr. 5th 2018

Online sketches ? Database Task Data Stream Maintain data sketch Learning Clustering Classification etc… = cat Data Stream … ? Idea! Maintain data sketch Distributed database Desired properties Fast to compute Preserve desired information Preserve data privacy Nicolas Keriven 1/9

Can we learn something more useful ? A very simple example Data Stream … What information ? Capture the mean of the data Memory use: Update cost: Sketch Can we learn something more useful ? Nicolas Keriven 2/9

Random features Data Stream What information ? … What information ? Capture generalized moments Example: histograms… Memory use: Update cost: Sketch In this talk… For the mathematicians… With high probability, approx. the kernel mean (Gretton 2008): contains all info. about distrib. of as random features (random projection + non-linearity) With LightOn optical technology: even for HD data Energy efficient Nicolas Keriven 3/9

Application 1: Sketch clustering Sketching Moment matching Theorem [Gribonval, Blanchard, K., Traonmilin 2017] With sufficiently many random features, with high probability the moment matching method is successful for K-means (mixtures of Diracs) Mixture models learning (GMM,…) Modelisation of the distribution of the data. Proof: information-preservation techniques on the probability distribution of the data. Infinite-dimensional analysis ! Greedy heuristic for moment matching: Compressive Learning OMPR (CL-OMPR) [Keriven 2016] 12/10/2017 Nicolas Keriven 4/9

Result: Compressive k-means [Keriven et al 2017] Mixture of Diracs = k-means Application: Spectral clustering for MNIST classification [Uw 2001] Classif. Perf. On this example… 1 order of magnitude faster than k-means 4 orders of magnitude more memory efficient Nicolas Keriven 5/9

Gaussian mixture models GMM d = 10, k = 20 Error Faster than EM (VLFeat’s gmm) Size of database Application: speaker verification [Reynolds 2000] (d=12, k=64) EM on 300 000 vectors : 29.53 20kB sketch computed on 50GB database: 28.96 Nicolas Keriven 6/9

Audio Source separation [Keriven et al 2018] Mixture of stable dist. Application: audio source separation Model: hybrid between rank-1 alpha-stable and Gaussian noise… Nicolas Keriven 7/9

Application 2: Online change point detection detect abrupt variations in a stream of data, by comparing two sketches. Help detecting extreme events, or selecting which data to store (segmentation) Applications in: Video monitoring Seismic data Speech detection … 12/10/2017 Nicolas Keriven 8/9

Application 2: Online change point detection : detect variations in the mean of : detect variations in the mean of = any change in distribution Theorem [Keriven2018] With sufficiently many random features, with high probability, the online change point detection procedure detects any significant change in the distribution of samples. Toy example: On a small laptop, no parallel computing, memory footprint < 5kb… 12/10/2017 Nicolas Keriven 8/9

Conclusion Online sketches: Data Stream … « D’ici 2040, les besoins en espace de stockage mondial […] risquent d’excéder la production disponible globale de silicium » Rapport Villani Online sketches: Handle streams of data, preserves data privacy Using random features: Capture all information about underlying probability distribution of data Can be computed extremely efficiently Nicolas Keriven 9/9

Thank you ! data-ens.github.io Keriven, Bourrier, Gribonval, Pérez. Sketching for Large-Scale Learning of Mixture Models Information & Inference: a Journal of the IMA, 2017. <arXiv:1606.02838> Keriven, Tremblay, Traonmilin, Gribonval. Compressive k-means ICASSP, 2017. Gribonval, Blanchard, Keriven, Traonmilin. Compressive Statistical Learning with Random Feature Moments. Preprint 2017. <arXiv:1706.07180> Keriven. Sketching for Large-Scale Learning of Mixture Models. PhD Thesis. <tel-01620815> Code: sketchml.gforge.inria.fr github: nkeriven data-ens.github.io Come to the colloquium! Come to the Laplace seminars! Nicolas Keriven