Privacy-Preserving Support Vector Machines via Random Kernels Olvi Mangasarian UW Madison & UCSD La Jolla Edward Wild UW Madison March 3, 2016 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A AAA A A A A A The 2008 International Conference on Data Mining

Horizontally Partitioned Data A A1A1 A2A2 A3A3 Data Features 1 2..………….…………. n Examples m m

Problem Statement Entities with related data wish to learn a classifier based on all data The entities are unwilling to reveal their data to each other If each entity holds a different set of examples with all features, then the data is said to be horizontally partitioned Our approach: privacy-preserving support vector machine (PPSVM) using random kernels –Provides accurate classification –Does not reveal private information

Outline Support vector machines (SVMs) Reduced and random kernel SVMs Privacy-preserving SVM for horizontally partitioned data Summary

K(x 0, A 0 )u =  1  Support Vector Machines K(A +, A 0 )u ¸ e  +e K(A , A 0 )u · e   e + _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ K(x 0, A 0 )u =  K(x 0, A 0 )u =  Slack variable y ¸ 0 allows points to be on the wrong side of the bounding surface x 2 R n SVM defined by parameters u and threshold  of the nonlinear surface A contains all data points {+…+} ½ A + {  …  } ½ A  e is a vector of ones SVMs Minimize e 0 s (||u|| 1 at solution) to reduce overfitting Minimize e 0 y (hinge loss or plus function or max{, 0}) to fit data Linear kernel: (K(A, B)) ij = (AB) ij = A i B ¢ j = K(A i, B ¢ j ) Gaussian kernel, parameter  (K(A, B)) ij = exp(-  ||A i 0 -B ¢ j || 2 )

Support Vector MachineReduced Support Vector Machine L&M, 2001: replace the kernel matrix K(A, A 0 ) with K(A, Ā 0 ), where Ā 0 consists of a randomly selected subset of the rows of A M&T, 2006: replace the kernel matrix K(A, A 0 ) with K(A, B 0 ), where B 0 is a completely random matrix Random Reduced Support Vector Machine Using the random kernel K(A, B 0 ) is a key result for generating a simple and accurate privacy-preserving SVM

Error of Random Kernels is Comparable to Full Kernels: Linear Kernels Full Kernel AA 0 Error Random Kernel AB 0 Error Each point represents one of 7 datasets from the UCI repository B is a random matrix with the same number of columns as A and either 10% as many rows, or one fewer row than columns Equal error for random and full kernels

Error of Random Kernels is Comparable Full Kernels: Gaussian Kernels Full Kernel K(A, A 0 ) Error Random Kernel K(A, B 0 ) Error

Horizontally Partitioned Data: Each entity holds different examples with the same features A1A1 A2A2 A3A3 A3A3 A2A2 A1A1

Privacy Preserving SVMs for Horizontally Partitioned Data via Random Kernels Each of q entities privately owns a block of data A 1, …, A q that they are unwilling to share with the other q - 1 entities The entities all agree on the same random basis matrix and distribute K(A j, B 0 ) to all entities K(A, B 0 ) = A j cannot be recovered uniquely from K(A j, B 0 )

B Privacy Preservation: Infinite Number of Solutions for A i Given A i B 0 Given – – Consider solving for row r of A i, 1 · r · m i from the equation –BA ir 0 = P ir, A ir 0 2 R n –Every square submatrix of the random matrix B is nonsingular –There are at least Thus there are solutions A i to the equation BA i 0 = P i If each entity has 20 points in R 30, there are solutions Furthermore, each of the infinite number of matrices in the affine hull of these matrices is a solution P ir A ir 0 = Feng and Zhang, 2007: Every submatrix of a random matrix has full rank

Results for PPSVM on Horizontally Partitioned Data Compare classifiers that share examples with classifiers that do not –Seven datasets from the UCI repository Simulate a situation in which each entity has only a subset of about 25 examples

Error Rate of Sharing Data is Better than not Sharing: Linear Kernels Error Without Sharing Data Error Sharing Data Error Rate Without Sharing Error Rate With Sharing 7 datasets represented by one point each

Error Rate of Sharing Data is Better than not Sharing: Gaussian Kernels Error Without Sharing Data Error Sharing Data

Summary Privacy preserving SVM for horizontally partitioned data –Based on using the random kernel K(A, B 0 ) –Learn classifier using all data, but without revealing privately held data –Classification accuracy is better than an SVM without sharing, and comparable to an SVM where all data is shared Related work –Similar approach for vertically partitioned data to appear in ACMTKDD –Liu et al., 2006: Properties of multiplicative data perturbation based on random projection –Yu et al., 2006: Secure computation of K(A, A 0 )

Questions Websites with links to papers and talks: