1. Tagging of digital historical images Authors: A. N. Talbonen (antal@sampo.ru) A. A. Rogov (rogov@psu.karelia.ru) Petrozavodsk state university

2. General tagging model Object selection Tag attribution Indexing Image collection Object DB FileTags I1…… I2…… Full-text index Tag DB

3. General research features  Research is based on analysis of image collection of White Sea-Baltic Sea Canal provided by National museum of Karelia  Collection consists of about 8k images with resolution 75 dpi.

4. 1. Face tagging General features  Predominance of small-sized objects (width is less than 40 pixels)  No database  Available expert Distribution of object’s size

5. 1. Face tagging General algorithm  Object (face) detection.  Computing of pairwise distances between objects.  Tagging (for each object): The system displays a list of the most similar objects. The expert determines a relationship between objects Object tags are specified

6. 1. Face tagging Face detection features  There is OpenCV library (OpenCvSharp in C#) and it’s method cv::CascadeClassifier::detectMultiScale (haarDetectObject in C#) (Viola-Jones implementation) being used for face detection  Viola-Jones method parameters are used to affect on precision and recall on face detection results  There is face recognition method based on Local Binary Patterns being used to improve the quality of Viola-Jones results

7. Training set Object Recognition Face objects Fake objects Object is a face Insert in result collection Yes Face detection Source image 1. Face tagging Face detection diagram

8. 1. Face tagging Local binary patterns (LBP) Original LBP filter Advanced LBP filters

9. 1. Face tagging Local binary patterns Uniform codes (patterns) Rotation invariant codes

10. 1. Face tagging Local binary patterns Weight matrix Computing of face object histogram

11. 1. Face tagging Face detection experiment  The purpose is to find the LBP modification with the best detection rates  Experiment features: Sample of 1070 images Assessing features  Fake object when: Object is not a face Faces are recognized weakly Faces turned at an angle greater than 90 degrees  Face object when: Object is a face Object is an image of people: portraits, paintings, sculptures 12 different LBP modifications were used

12. 1. Face tagging Face detection experiment results

13. 1. Face tagging Face recognition experiment  Purpose is to find the LBP modification with the best face recognition rates  Experiment features Training set contains 19 objects including 3 relevant pairs of face objects and 1 relevant pair of fake objects 10 LBP modifications were used

14. 1. Face tagging Face recognition experiment 1)2)3)4)5) 6)7)8)9)10) 11)12)13)14)15) 16)17)18)19) Pairs: {1, 15}, {3, 14}, {4, 13}, {7, 9}

15. 1. Face tagging Face recognition experiment results Взвешенный ModificationPrecision 0,38 0,25 0,50 0,75 0,50 0,38 0,63 1,00

16. 1. Face tagging Face comparing Training set object’s histograms: Objects at position (row, col): (1,1) and (3, 4) correspond to fake objects and have similar histograms very different from the rest

17. 2. Texture tagging General features  The classifier with tags based on moments is built  Texture searching is based on the built classifier  Search involves finding the segments corresponding to different textures  Minimal segment size to be include in result is 100 pixels

18. 2. Texture tagging Moment-based segmentation Moment calculation function: Source image IMoment image M00 Moment image M10Moment image M01

19. 2. Texture tagging Moment-based segmentation F00 F10 F01 Binary segmentation example Precision: 96,7 % Moment feature calculation function:

20. 2. Texture segmentation Implementation features  Each moment is an image  Moment computing is based on library OpenCV and it’s method cv::filter2D  Parameter seek is based on developed experiment

21. 2. Texture tagging Parameter seek example Moment window size Moment feature Window size SigmaPrecision 9490,0195,285 9390,00595,1782 9390,0295,1752 9440,00595,1355 9490,01595,1324 14 0,0293,8416 14 0,00593,7103 14190,00592,7826 14190,01592,7826 14340,01592,5293 14290,01592,395 14340,0292,3248 24 0,0287,9639 39190,0187,9639

22. 2. Texture tagging Classifier features  Set of textures of several classes is given  Each class is assigned a set of tags  Each image is subjected to a separate texture search  Each texture found adds appropriate set of tags to the source image

23. 2. Texture tagging Example Source image

24. 2. Texture tagging Example Classifier example Classifier textures example

27. 2. Texture tagging Experiment  Purpose is to evaluate the search quality  Experiment features Sample of 100 images Classifier contains 2 textures House roofHouse wall

28. 2. Texture tagging Search quality evaluate method - Flag of belonging to assessed collection - Flag of belonging to search result Flag of relevance Single texture estimations: General estimations:

29. 2. Texture tagging Experiment results

30. Thanks for your attention!