1. Semantic Tolerance Relation-based Image Representation and Classification
Ying Dai
Faculty of software and information science,
Iwate Pref. University
2019/2/17

2. Outline Background Architecture of large image/video retrieval
Mechanism of Image/key-frame semantic representation
Algorithm for automatic representation of image/key-frame semantics
Experiments and analysis
2019/2/17

3. Background
More and more people express themselves by sharing images, video on line
However, It is still hard to manipulate, index, or search through them, because of the semantic gap between user and machine
Machine retrieving feature-similar contents
User retrieving semantic-similar contents, following color, structural similarity.
The goals to developing systems will focus on
understanding images and
Presenting their content in a form intuitive to humans by machine
With the technological advances in digital imaging, networking, and data storage, more and more people communicate with one other and express themselves by sharing images, video and other forms of media on line. However, it is still hard to manipulate, index, filter, summarize, or search through them, because of the semantic gap between user and machine, which means that there are many queries for which visual similarity does not correlate strongly with human similarity judgments, because machine retrieves feature-similar contents, but human retrieves semantic-similar contents, following color, structural similarity . Therefore, developing systems capable of understanding images and able to represent their content in a form intuitive to humans becomes one of most motivate things in the filed of large image/video retrieval.
2019/2/17

4. Architecture of large image/video retrieval (1)
by the local content storing system, each image/video is represented by its semantics and low-level features, such as color, shape, and texture. Then, by the P2P network-based index sharing system, the similar image/video is retrieved according to the domain/category-based query and sample-based query. Moreover, because the interpretation of finding similar images is ambiguous and subjective on the level of human perception for different users, the extraction of individual tolerance criteria to the similarity is needed in the process of retrieving image/video. How to extract low-level features and retrieve visually similar images by a sample-based query was introduced in author’s previous papers. In this paper, we mainly introduce how to consider the semantic tolerance relation between categories, and how to assign semantics to each image/video automatically.
2019/2/17

5. Mechanism of Image/key-frame semantic representation
Semantic tolerance relation model
because the concepts of image/video in many domains are imprecise, and the interpretation of finding similar image/video is ambiguous and subjective on the level of human perception, we define the semantic categories of image and key-frame, together with the tolerance degrees between them.
Images are described by different domains. For a certain domain, concepts are divided into some classes. The class may be associated with the other in a same domain. The relation between them is called as intra-association. Also, the class may be associated with the other in the different domains. The relation between them is called as inter-association.
2019/2/17

6. Semantic tolerance relation model (1)
Class co-occurrence matrix
Regarding the number of images which are both assigned to class i and class j.
Tolerance degree between two classes
Semantic tolerance relation model (STRM): a matrix regarding tolerance degree between classes
In order to generate the semantic tolerance relation model, firstly, class co-occurrence matrix is introduced. It is a matrix regarding the number of images which are both assigned to class i and class j..
Then the Tolerance degree between two classes can be defined as:
Therefore, Semantic tolerance relation model is a matrix regarding tolerance degree between classes
2019/2/17

7. Semantic tolerance relation model (2)
for the nature vs. man-made domain with 7 classes: patterns, flower arrangement, tree & plant, sunset, building, and food & tableware,
2019/2/17

8. Semantic representation and categorization (1)
Each image/key-frame is represented by a vector of classes’ weights based on STRM
where
un-tolerance parameters to eliminate the influence of the false tolerance caused by the training image sets.
are un-tolerance parameters
2019/2/17

9. Semantic representation and categorization (2)
For the image categorization regarding single domain
For the image categorization regarding cross-domains
For the image categorization regarding single domain , images are grouped to a class i , when the values of element i regarding their representing vectors are larger than a clustering threshold ,
For the image categorization regarding cross-domains, the grouped images are the intersection of those which either belong to the class i in domain k, or belong to the class j in domain l.
2019/2/17

10. Algorithm for automatic presentation (1)
In the pre-processing procedure, the semantic tolerance relation model is created by using the classification results of training images set. In the image/video indexing procedure, key-frame is firstly extracted from the videos. Then, in order to realize the automatic representation of image/key-frame semantics based on the semantic tolerance relation model, it is necessary to extract the low-level features of image/key-frames, and classify each image using the designed classifier. According to the classification result, each image/key-frame is semantic-represented by a vector of the classes’ weights.
2019/2/17

11. Algorithm for automatic presentation (2)
any algorithms for the feature extraction and the classification can be used
For representing images regarding nature vs. man-made domain
SGLD matrices of H, S, V components is used to extract low-level features
Bayesian classifier is used to classify each image
For representing images regarding human vs. non-human domain
Skin color is used to judge whether there are faces and the size of them.
2019/2/17

12. Examples of categorization
Some images categorized to the group of persons with the background of food & tableware
2019/2/17

13. Experiment and analysis
Precision-recall of the classes of patterns and landscape
compared with a single precision-recall when STRM is not used, STRM’s conducting improves the performance of image categorization.
Moreover, STRM’s conducting enables the dynamic association of precision and recall, which means that the precision and recall of image categorization can be controlled according to user’s requirements, and make the image categorization with more flexibility.
2019/2/17

14. Comparison to the state-of-the-art
method
Defined classes
Precision/recall
objects
relation
number
Scalable
dynamics
rate
This
paper
nature vs. man-made,
human vs. non-face,
time, location, and so on
tolerance-
considered
multi-
yes
0.88/0.85
M.R.Bo
indoor/outdoor, or
nature/man-made
opposite
binary- no --
X. Shen
scene
tolerance-considered
0.69/0.85
A. Vai.
outdoor
orientated
0.89/0.76
J.Yu
face/non-face
0.86/0.85
2019/2/17