Presentation is loading. Please wait.

Presentation is loading. Please wait.

Fast Kernel Methods for SVM Sequence Classifiers Pavel Kuksa and Vladimir Pavlovic Department of Computer Science Rutgers University.

Similar presentations


Presentation on theme: "Fast Kernel Methods for SVM Sequence Classifiers Pavel Kuksa and Vladimir Pavlovic Department of Computer Science Rutgers University."— Presentation transcript:

1 Fast Kernel Methods for SVM Sequence Classifiers Pavel Kuksa and Vladimir Pavlovic Department of Computer Science Rutgers University

2 Outline Problem & Motivation: Barcode of Life Spectrum kernels: exact and mismatch kernels Full spectrum kernel algorithms  Exact spectrum kernel algorithms  Adding mismatches: incremental mismatchkernel computations Sparse Spectrum kernels & algorithms Algorithmic complexity Experimental results Conclusions

3 Problem & Motivation Barcoding of Life: classification and identification of organisms using ’barcodes’ DNA barcode: a short fragment (≈600bp) from standard mitochondrial gene region assign an unidentified sample to the taxonomic group at a target level (class, family, species, etc.)using reference data with known class information Supported by many organizations (CBOL, BOLD) Applications: biodiversity monitoring and assessment, ecological studies, taxonomic research, etc.

4 Problem & Motivation Methods widely used: kernel-based machine learning methods  most accurate results in many sequence analysis and prediction tasks  often computationally expensive This work: general spectrum-like string kernel methods with the focus on DNA-based sequence analysis Goal: a fast and accurate classification:  small space, fast to estimate, accurate results

5 Barcode-based species identification Problem: Given a reference set R of barcodes from different species S identify, for a newly obtained barcode b, its category or decide that it is of new category Identification at the species/sub-species level Multiclass classification with species as target classes, no phylogenetic analysis and/or higher level classification Solve multiple binary membership problems using kernel-based classifiers

6 Kernel methods for sequence analysis Fisher kernel: a profile HMM is built for each class and then used as a feature extractor Alignment-based kernels: measure mutual distances between sequences or their profiles  local alignment (LA) kernels  SVM-pairwise, etc.  extremely time consuming Spectrum-like string kernels Many others!

7 String spectrum kernels General idea:  sequence features (e.g. short substrings or certain patterns) to compute histograms of feature frequencies (spectrum)  compare the feature histograms  sequences with similar content have similar spectra Examples:  the exact spectrum kernel: counts fixed- length substrings  the spectrum kernel with mismatches:  in addition to the explicit substrings, counts also similar substrings (neighbors)  less sparse spectrum

8 The exact spectrum kernel The k-spectrum of a sequence x is the set of all its substrings of length k Dimensionality of the k-mer feature space is |Σ| k, where |Σ| is the size of alphabet The k-spectrum map Φ k (x):  Φ k (x) : x → [φ k a (x)] ∀ a ∈ Σ k  φ k a (x) = | {j : x(j : j − k + 1) = a} | The k-spectrum kernel: K k (x, y) =

9 Example 3-spectrum of ’ACGAC’: 4 3 = 64 vector Φ 3 (x) = [AAA/0,..., ACG/1,..., CAC/1,..., CGA/1...]

10 Mismatch kernel Two integer parameters: k, and m Parameter m is the maximum number of the character mismatches between two k- substrings The mismatch feature map: φ (k,m) a (x) = |{j : d(x(j : j + k − 1), a) ≤ m, a ∈ |Σ| k }| The mismatch kernel: K k,m (x,y) = (k, m)-neighborhood size v = Σ m i=0 C(k,i) (|Σ| − 1) i

11 Spectrum kernel computations Single kernel computations: compute kernel K(x, y) for two sequences Kernel matrix computations: compute NxN matrix for N sequences Kernel vector computations: compute Nx1 kernel vector for sequence x and N support sequences Focus on kernel matrices and vectors (SVM training, evaluation/testing, clustering, etc.)

12 Exact spectrum kernel computations Suffix tree-based algorithms:  for each pair (x, y) build a suffix tree for sequence features, and compute the kernel K k (x, y) at the tree leaves  O(N 2 kn) for N sequences of length n each Explicit mapping (EMap) algorithms  explicitly map sequences to feature vectors of size |Σ k |, kernel matrix is then computed as K = M.M’, where M N×|Σ| k is a feature matrix  O(Nkn + |Σ k |N 2 ) Sorting-based algorithms:  Represent k-mers as numbers and use favorite sorting algorithm

13 Counting-sort spectrum kernel Algorithm: 1. Extract k-length substrings from input sequences and store them in a list L, O(knN ) 2. Sort the list L using k passes of counting sort, O(knN ). 3. Scan the sorted list updating the kernel matrix on each change in the feature value Complexity: O(Nnk + min(u, n)· N 2 ), u - number of unique k-mers in the input Update step:  K(upd f, upd f ) = K(upd f, upd f ) + c f c f T  upd f = {i : f ∈ x i } = input sequences containing f  c f = vector of feature counts

14 Mismatch kernel using sorting Sorting-based computations:  Extract unique features using sorting  Expand a set of unique features to include their neighbors  Sort the resulting set  Scan the sorted list and update kernel matrix on each change in the feature value Complexity: O(Nnk + uvk + uN 2 )

15 Divide-and-Conquer Mismatch Kernel Basis idea:  Infer count for a k-mer f using counts of its neighbors  Cluster the combined feature set S = ∑ N i=1 Spectrum(x i ) to find sets of neighboring features  The size of resulting clusters/subclusters gives desired counts of feature occurrences For DNA, since u << nN, improve performance by using a unique features instead of the original redundant set.

16 Divide-and-Conquer Method Divide step: the combined feature set S is partitioned into subsets S 1,..., S |Σ| using character-based clustering Conquer step: The same procedure (Divide step)is applied to each of the obtained subsets. After k divisions, kernel matrix is updated according to the contribution of the corresponding k-mer f. Complexity:

17 Sparse kernel Can we fur ther reduce computation costs?  Preselect features (e.g. using filtering) and evaluate kernel for a set F of selected features  K = M F T M F, M F is |F|×N matrix of feature counts Reduces complexity of computations:  Spectrum kernel: O(Nnk + FN 2 ) vs O(Nnk + uN 2 )  Mismatch kernel: O(Nnk + Fvk + FN 2 ) vs O(Nnk + uvk + uN 2 )

18 Complexity comparison: Spectrum Previously known bounds: O(knN 2 + nN 2 ) New bounds: O(knN + uN 2 ) Advantages of counting sort-based computations:  more time efficient  smaller memory requirements in practice than suffix trees  easier to implement

19 Complexity comparison: Mismatch Previously known bounds: O(nk m+1 |Σ| m N 2 ) New bounds: O(uk m+1 |Σ| m + uN 2 ) EMap=explicit map, EMap+Sort=EMap with presorting, DC=divide and conquer v=neighborhood size, u=number of different k-mers in the input, u′=number of different k-mers including neighbors

20 Experimental framework Barcode datasets Classification: multiclass & binary (CV/ROC/ROC50) Kernels: Fisher, Spectrum, Mismatch Algorithms: SVM, ridge regression, 1-NN Running time analysis:  training (matrix) and testing (vector)  different kernel parameters: k, m  different feature selection levels

21 Classification performance: Multiclass 10-fold cross-validation MK=mismatch kernel, SK=spectrum kernel, SMK, SSK=with feature selection, NN=nearest neighbor, FK=Fisher kernel 10% features, classifiers improve/retain performance Improved performance compared to previous studies in [Matz & Nielsen, 2005] and [Nielsen & Matz, 2006](rates of 9-20%) %Error

22 Classification performance: Binary 10-fold cross-validation error rates Average ROC/ROC50 scores

23 Classification performance: ROC Feature selection improves performance 90% reduction in the number of features AstraptesHesperiidae

24 Running time: Mismatch Running time, seconds Significant time improvement compared to the state-of-art spectrum kernel implementation EMap requires much larger storage than D&C Pre-sorting significantly improves computing time for EMap

25 Mismatch + Feature Selection D&C scales almost linearly with the number of features

26 Running time: Mismatch vector D&C outperforms EMap in many cases while requiring only linear space

27 Summary of results Efficient computation of spectrum kernel matrices and vectors: counting sort and divide- and-conquer techniques Spectrum kernels for accurate and fast DNA barcode-based species identification Few sequence features can successfully discriminate species Small discriminative subsets of k-mers (signatures)for many taxonomic groups

28 Future work Position-aware string kernels: taking feature interactions into account Smoothed kernels (independent of a choice of k) Efficient feature selection methods Learning low-dimensional representations Semi-supervised setting Direct multiclass methods

29 References [Kuang 04] Rui Kuang, Eugene Ie, Ke Wang, Kai Wang, Mahira Siddiqi, Yoav Freund & Christina Leslie. Profile-Based String Kernels for Remote Homology Detection and Motif Extraction. In CSB ’04: Proceedings of the 2004 IEEE Computational Systems Bioinformatics Conference (CSB’04), pages 152–160, Washington, DC, USA, 2004. IEEE Computer Society. [Leslie 02a] Christina S. Leslie, Eleazar Eskin & William Stafford Noble. The Spectrum Kernel: A String Kernel for SVM Protein Classification. In Pacific Symposium on Biocomputing, pages 566– 575, 2002. [Leslie 02b] Christina S. Leslie, Eleazar Eskin, Jason Weston & William Stafford Noble. Mismatch String Kernels for SVM Protein Classification. In NIPS, pages 1417–1424, 2002. [P.D.N. 03] Heber t P.D.N., A. Cywinska, S.L. Ball & J.R. deWaard. Biological identifications through DNA barcodes. In Proceedings of the Royal Society of London, pages 313–322, 2003. [Vishwanathan 02] S. V. N. Vishwanathan & Alexander J. Smola. Fast Kernels for String and Tree Matching. In NIPS, pages 569–576, 2002.


Download ppt "Fast Kernel Methods for SVM Sequence Classifiers Pavel Kuksa and Vladimir Pavlovic Department of Computer Science Rutgers University."

Similar presentations


Ads by Google