Tagging of digital historical images Authors: A. N. Talbonen A. A. Rogov Petrozavodsk state university

General tagging model Object selection Tag attribution Indexing Image collection Object DB FileTags I1…… I2…… Full-text index Tag DB

General research features  Research is based on analysis of image collection of White Sea-Baltic Sea Canal provided by National museum of Karelia  Collection consists of about 8k images with resolution 75 dpi.

1. Face tagging General features  Predominance of small-sized objects (width is less than 40 pixels)  No database  Available expert Distribution of object’s size

1. Face tagging General algorithm  Object (face) detection.  Computing of pairwise distances between objects.  Tagging (for each object): The system displays a list of the most similar objects. The expert determines a relationship between objects Object tags are specified

1. Face tagging Face detection features  There is OpenCV library (OpenCvSharp in C#) and it’s method cv::CascadeClassifier::detectMultiScale (haarDetectObject in C#) (Viola-Jones implementation) being used for face detection  Viola-Jones method parameters are used to affect on precision and recall on face detection results  There is face recognition method based on Local Binary Patterns being used to improve the quality of Viola-Jones results

Training set Object Recognition Face objects Fake objects Object is a face Insert in result collection Yes Face detection Source image 1. Face tagging Face detection diagram

1. Face tagging Local binary patterns (LBP) Original LBP filter Advanced LBP filters

1. Face tagging Local binary patterns Uniform codes (patterns) Rotation invariant codes

1. Face tagging Local binary patterns Weight matrix Computing of face object histogram

1. Face tagging Face detection experiment  The purpose is to find the LBP modification with the best detection rates  Experiment features: Sample of 1070 images Assessing features  Fake object when: Object is not a face Faces are recognized weakly Faces turned at an angle greater than 90 degrees  Face object when: Object is a face Object is an image of people: portraits, paintings, sculptures 12 different LBP modifications were used

1. Face tagging Face detection experiment results

1. Face tagging Face recognition experiment  Purpose is to find the LBP modification with the best face recognition rates  Experiment features Training set contains 19 objects including 3 relevant pairs of face objects and 1 relevant pair of fake objects 10 LBP modifications were used

1. Face tagging Face recognition experiment 1)2)3)4)5) 6)7)8)9)10) 11)12)13)14)15) 16)17)18)19) Pairs: {1, 15}, {3, 14}, {4, 13}, {7, 9}

1. Face tagging Face recognition experiment results Взвешенный ModificationPrecision 0,38 0,25 0,50 0,75 0,50 0,38 0,63 1,00

1. Face tagging Face comparing Training set object’s histograms: Objects at position (row, col): (1,1) and (3, 4) correspond to fake objects and have similar histograms very different from the rest

2. Texture tagging General features  The classifier with tags based on moments is built  Texture searching is based on the built classifier  Search involves finding the segments corresponding to different textures  Minimal segment size to be include in result is 100 pixels

2. Texture tagging Moment-based segmentation Moment calculation function: Source image IMoment image M00 Moment image M10Moment image M01

2. Texture tagging Moment-based segmentation F00 F10 F01 Binary segmentation example Precision: 96,7 % Moment feature calculation function:

2. Texture segmentation Implementation features  Each moment is an image  Moment computing is based on library OpenCV and it’s method cv::filter2D  Parameter seek is based on developed experiment

2. Texture tagging Parameter seek example Moment window size Moment feature Window size SigmaPrecision 9490,0195, ,00595, ,0295, ,00595, ,01595, ,0293, ,00593, ,00592, ,01592, ,01592, ,01592, ,0292, ,0287, ,0187,9639

2. Texture tagging Classifier features  Set of textures of several classes is given  Each class is assigned a set of tags  Each image is subjected to a separate texture search  Each texture found adds appropriate set of tags to the source image

2. Texture tagging Example Source image

2. Texture tagging Example Classifier example Classifier textures example

2. Texture tagging Experiment  Purpose is to evaluate the search quality  Experiment features Sample of 100 images Classifier contains 2 textures House roofHouse wall

2. Texture tagging Search quality evaluate method - Flag of belonging to assessed collection - Flag of belonging to search result Flag of relevance Single texture estimations: General estimations:

2. Texture tagging Experiment results

Thanks for your attention!