1. Gene selection using Random Voronoi Ensembles Stefano Rovetta Department of Computer and Information Sciences, University of Genoa, Italy Francesco masulli Department of Computer Science, University of Pisa, Italy

2. The input selection problem Hard Given d inputs, there are 2 d possible subsets and no guarantee that larger subset perform better/worse than smaller (a.k.a.: no monotonicity) Classic A lot of references dating back from about mid-seventies Important Curse of dimensionality, Generalization, Cost of measurements, Cost of computation...

3. A different perspective Although old, the input selection problem is being actively studied now From optimization Classic approach: improve training speed / generalization ability / computational resources requirements......to model analysis Mainstream approach as of today: find the subset of inputs which account the most for the observed phenomenon A tool for scientific inquiry, not for system design

4. Gene selection Bioinformatics is where input selection is a current (hot) topic DNA microarrays provide bulks of simultaeous data – e.g., gene expression We have to find out which genes are the most relevant to a given pathology (Good candidates to be the true cause) We are interested in a specific approach: assessing the relative importance of each input variable (gene)

5. Problem statement We address: – Classification problems – with 2 classes only to simplify the analysis (can be extended to multiclass) – seeking a saliency ranking - on a d-dimensional vector space: x   d A single separating function is assumed, denoted by g(x)

6. Outline of the technique The proposed technique has three components 1 – a local analysis step with a basic classifier 2 – a resampling procedure to iterate step 1 3 – an integration step

7. Saliency (or importance or sensitivity or...) Many definitions Intuitively: some attribute of an input variable which measures its influence on the solution of a given (classification) problem The derivative of the output w.r.t. each input variable is a natural measure of influence  g(x) = (∂g(x)/∂x 1,..., ∂g(x)/∂x d ) But...

8. Finite sample effects The rule is learned from a training set: random variability Derivatives and local fluctuations often it is better to study difference ratios ( f(x+Δ) – f(x) ) / Δ rather than derivatives f'(x)

9. Use of linear separators If the decision function is of the form g(x) = w. x then derivatives w.r.t. inputs are constant and given directly by the coefficient vector w SVMs can provide the optimum linear separators w.r.t. a given generalization bound 2-norm soft margin optimization: bound on generalization error based on (soft) margin such linear separators are robust in terms of sample variations (they depend on support vectors only)

10. Local analysis The linear separator is applied on a local basis Nonlinear g(x) can be studied by local linearization Voronoi partitioning A Voronoi tessellation is performed on the training set Linear analysis is applied within each Voronoi polyhedron (a localized subset of training samples) We obtain a saliency ranking directly by t = w/max{w i } (signs can be discarded and analyzed separately)

11. Drawbacks Several: mainly border effects and small sample size within Voronoi polyhedra Solution: resampling The Voronoi tessellation is performed several times Random Voronoi tessellations are used each time

12. An ensemble method The procedure can be seen as an ensemble of localized linear classifiers The necessary classifier diversity is provided by random Voronoi tessellations What we need next: Integration of local analyses

13. Integrating by clustering For each Voronoi polyhedron of each resampling step, we obtain a pair of d-dimensional vectors (or a 2d- dimensional combined vector) v i = ( t i, y i ) where: t i the saliency ranking y i the Voronoi centroid (site) To integrate the local analyses we perform a c-means type clustering on vectors v i

14. Some details on the clustering step - The clustering technique is the Graded Possibilistic c-Means algorithm - The dimensionality problem is easily tackled by working only within the subspace spanned by the training set - Clusters are obtained by merging (averaging) sets of vectors v i which are close either by their y (location) or by their t (saliency pattern) components - The number of clusters is currently to be prespecified (as in standard c-means) It is independent on the number of voronoi sites used

15. Results “Leukemia” data set by Golub et al.

16. Discussion and future work The results indicate that some of the genes indicated by the original work by Golub et al. are found to be important also by our approach. Extensive validation (by the help of domain experts or biologists) must be done The direction (sign) of saliency has been found to be always in agreement with statistical correlation as indicated by the original work. Further experiments: a new data set (still unpublished) is currently being investigated An interesting tweak: replacing the general c-means-type clustering with a technique specifically tailored on rank data