1. 1 E.V. Myasnikov 2007 Digital image collection navigation based on automatic classification methods Samara State Aerospace University RCDL 2007Интернет-математика 2007 mevg@smr.ru Навигация по коллекциям цифровых изображений на основе методов автоматической классификации Самарский государственный аэрокосмический университет Е.В. Мясников

2. 2 Navigation in collection of digital images alternative to image retrieval system complement to image retrieval system convenient browsing system Approaches to navigation system construction to construct projection of the whole image collection into 2-D navigation space to cluster image collection into the set of clusters (hierarchy) and then construct 2-D projection of each cluster to construct tree-like structure using an optimization rule

3. 3 Clustering methods Hierarchical clustering (agglomerative) Single link Complete link Average link Nonhierarchical clustering K-means Kohonen neural networks (SOM) Fuzzy clustering

4. 4 Linear Principal component analysis (PCA) Nonlinear Classical Kruskal MDS (multidimensional scaling) Sammon projection Force-Directed Replacement Projection methods Discrete lattice solution Continuous solution

5. 5 Demands to the navigation system Representation of the collection has a form of 2D vectors (as icons, points on the monitor) The set of images having higher level of similarity is displayed when bringing near the region The set of images having lower level of similarity is displayed when moving away from the current region Property of reversibility Operations with the navigation “map” Scrolling (up, down, left, right) Scaling (up, down)

6. 6 Main phases of proposed approach Feature extraction Cluster hierarchy construction Mapping into 2-D navigation space using restrictions imposed by cluster hierarchy Digital images Navigation space

7. 7 Clustering Phase: Analyzed Methods Hierarchical clustering scheme 1.Adjacency matrix calculation 2.Rank each object among clusters 3.Merge elements with minimal distance between them 4.Elimination of the raw and column of absorbed cluster and matrix recalculation 5.Stopping criterion test and transition to the step 3 Inter-cluster distance single link minimal distance between objects involved in clusters complete link maximal distance between objects involved in clusters Kohonen neural network WTA correction rule: w (t+1) = w (t) +  (t)[x(t) - w (t)] d(x(t), w (t)) = min 1  i  K d(x(t), w i (t)) Following equation holds true for the winning neuron To construct the hierarchy of clusters Kohonen neural network functions in a recursive order

8. 8 Clustering: Experimental results * Experiment was conducted on samples of size equal to 1000 Number of clusters Average quantization error * Single link Complete link WTA 250.3870.1930.169 500.3590.1500.139 1000.2930.1160.112 Quantization error: Examples of clusters

9. 9 Mapping Phase: Sammon projection Error d ij - distance between objects i and j in multidimensional space d * ij - distance between objects i and j in two dimensional space y jk - coordinates in 2D space Iterative formula Notation Operational time ~ O[N 3 ] (under the assumption that the number of iterations is of the same order as the number of objects)

10. 10 Construction of initial configuration for Sammon mapping Average error value * Number of iterations 100200300 Sammon mapping with random initalization 0.1260.0780.056 Best Sammon mapping over 10 runs with random initialization 0.0930.0510.035 PCA0.139 Two-phase method (PCA as initial configuration for Sammon mapping) 0.0440.0410.039 * samples of 100 images from dataset of 10 000 images were used to conduct the experiment Two-phase method example

11. 11 Methods of speeding-up Sammon projection 1. Triangulation 2. Neural Network 3. Approximation using random sets Chalmers’96 adaptation for Sammon projection (CS) Two sets are constructed for each object on each iteration: set of k 1 close objects set of k 2 random objects Operational time ~ O[N 2 ] (under the assumption that the number of iterations is of the same order as the number of objects and k 1 +k 2 << N)

12. 12 Proposed Methods: Combined Method (CM) 1.Build Sammon projection for the top level of the cluster tree 2.Build Sammon projection for the each subcluster at level 2 using 2D coordinates of the superclasters as fixed points 3.Repeat the process for each subclaster (or object) of the level 3 and so on Idea : Use hierarchical clustering to build the projection Method description Operational time – for balanced tree with depth L Modification of method (MCM): Use 2D coordinates of top level clusters for each subcluster (or object) at any level Special case: O[N 2 ] – for balanced tree with depth 2

13. 13 Proposed Methods: Restrictive Combined Method (CMR) 1.Map centers x u 1 of top level clusters С u 1  C 0 to the 2D vectors y u 1 using dimensionality reduction method (Sammon or two-phase method). Set boundaries of the whole displayed area  0 =  2.For each cluster С u k  C v k-1 of the current level k carry out points 3-6 3.Construct boundaries  u k of the cluster C u k in 2D space using centers coordinates y m k in 2D space of the clusters C m k, m=1..|C v k-1 | of the current level k 4.Complete cluster boundaries  u k using boundaries  v k-1 of the parent cluster C v k-1 at the previous level:  u k =  u k   v k-1 5.Map centers x i k+1 of all subclusters С i k+1  С u k (or immediately images O i ) at level k+1 to 2D vectors y i k+1, using boundaries  u k of the cluster applying the following recurrence relation 6. Apply described in points 3-5 procedure to map child clusters C i k+1 in the recursive order

14. 14 Proposed Methods: Modifications for CMR Function 1. Full correction rule (CMR-1) – if y i exceeds the bounds of the cluster then the correction value ensure y i to be on the boundary at the next step 2. Piece-wise linear rule (CMR-2) – correction value ensure the “attraction” to the center of the cluster or to the boundary when y i comes near or exceeds the boundary can be selected based on minimization of functional consisted of Sammon error and boundary function Two models were considered Example of CMR-1

15. 15 Experimental Research METHOD 1000 images 5000 images Average error value Mean square deviation of error Average operation time Average error value Mean square deviation of error Average operation time PCA0.11710.0144920.11480.00969011 CS0.028800.005668620.025920.0016571880 CM0.034070.002638170.062200.02875631 MCM0.027670.002047190.028400.00196267 CMR-10.029720.002218130.031430.00189031 CMR-20.034940.002590600.037230.002076276

16. 16 Example of MCM sample size: 10 000 images

17. 17 Example of CMR sample size: 10 000 images

18. 18 Example of navigation (CMR) region “а” region “б”

19. 19 region “в” region “г” Example of navigation (CMR)

20. 20 Selection of features Note: “Measures that are more effective for retrieval tend to be more complex, and thus lose their advantage over the simpler measures when forced into two dimensions” (K.Rodden, W.Basalaj, D.Sinclair, K.Wood A comparison of measures for visualising image similarity. In The Challenge of Image Retrieval. British Computer Society Electronic Workshops in Computing, 2000) Features:Color histograms in CIE L*a*b color space Metrics:Euclidian Main requirement to the feature system: Configuration of images in navigation space must be understandable to user

21. 21 The requirements to the navigation method are considered Novel navigation method is proposed Novel combined method and its modifications for dimensionality reduction are proposed Proposed methods are compared to known method The results of experimental analysis of methods being used are present Conclusions Future plans Exploring new feature systems Method improvement Estimation of effectiveness of navigation method including expert estimation

22. 22 This work was financially supported by Yandex (www.yandex.ru) The dataset “Image database” was provided by Yandex Acknowledgements THANK YOU FOR YOUR ATTENTION