1. Image/Video’s Semantic Assignment Using Bidirectional Associative Memories
Ying Dai
Faculty of software and information science,
Iwate Pref. University
2019/2/23

2. Outline Background class-based image/key-frame representation
Semantic similarity
Visual similarity
Associative values with pre-defined classes
Calculation of SS, VS and associative values
Bidirectional associative memories
Image/key-frame retrieval
Experiments and analysis
2019/2/23

3. Background
More and more people express themselves by sharing images and videos on line
It is still hard to manipulate, index, or search on-line images and videos
The objectives to our research focus on
understanding image/videos’ semantics by machine
representing image/videos’ semantics in a form intuitive to humans
With the technological advances in digital imaging, networking, and data storage, more and more people communicate with one other and express themselves by sharing images, video and other forms of media on line. However, it is still hard to manipulate, index, filter, summarize, or search through them, because of the semantic gap between user and machine, which means that there are many queries for which visual similarity does not correlate strongly with human similarity judgments, because machine retrieves feature-similar contents, but human retrieves semantic-similar contents, following color, structural similarity . Therefore, developing systems capable of understanding images and able to represent their content in a form intuitive to humans becomes one of most motivate things in the filed of large image/video retrieval.
2019/2/23

4. Category similarity correlation of image/key-frame (1)
　Image/key-frames are described by different domains, such as temporal, spacial, impression, nature vs. man-made, human vs. non-human, copyright, etc.
　Divided classes can be inter-correlation or intra-correlation
　the correlation is measured by two indices:
semantic similarity (SS);
visual similarity (VS)
because the concepts of image/video in many domains are imprecise, and the interpretation of finding similar image/video is ambiguous and subjective on the level of human perception, we define the semantic categories of image and key-frame, together with the tolerance degrees between them.
Images are described by different domains. For a certain domain, concepts are divided into some classes. The class may be associated with the other in a same domain. The relation between them is called as intra-association. Also, the class may be associated with the other in the different domains. The relation between them is called as inter-association.
2019/2/23

5. Category similarity correlation of image/key-frame (2)
Based on the fact that people judge image similarity by semantic similarity (SS), following visual similarity (VS).
Category similarity correlation indices
two facts of SS and VS, depicted by
: SS index, generated by the knowledge of subjects
　　　　　: VS index, calculated based on the learned samples’ visual features
All of them in the range of [0,1]
2019/2/23

6. Generation of SS correlation value
Class co-occurrence matrix
Regarding the number of images which are both assigned to class i and class j.
Semantic similarity degree between two classes
Semantic similarity correlation index
2019/2/23

7. Generation of VS correlation value
Bidirectional associative memories (by B.Kosko)
Associating tow patterns (X,Y) such that when one is encountered, the other can be recalled.
Storing the associated pattern pairs by connection weight matrix 1
W11
Wmn Y X1 Xm Ｘ
2019/2/23

8. Generation of VS correlation value
Bidirectional associative memories
Input and output in the X layer and Y layer
The energy of the BAM decreases or remains the same after each unit update.
BAM eventually converge to a local minimum that corresponds to a stored associated pattern pair.
2019/2/23

9. Stored pattern images of X layer (examples, nature vs. man-made domain)
Football (P1)
Landscape (P2)
flower (P3)
Tree (P4)
Flower (painting ) (P5)
Food (P14)
Building (P6)
Restaurant (P8)
Wood (P10)
Paper (P11)
2019/2/23

10. Stored patterns of Y layer
Encoding units
Particular unit outputs (recall values)
Stored patterns of Y layer are the vectors of I components which reprsent the encoding units of classes. The vector has I components if I classes are pre-defined.
2019/2/23

11. Generation of VS correlation value
VS correlation index
Let a pattern image represent a class, and I classes are pre-defined. For a class i with a pattern image i, VS to the class j
2019/2/23

12. Some examples of VS correlation value1
0.25
0.74
0.16
0.33
0.53
0.37
0.22
0.5
0.08
0.31 2
0.21
0.28
0.24
0.02
0.2
0.14 3
0.36
0.38
0.05
0.82
0.59
0.19
0.35
0.61
0.23 4
0.26
0.3
0.94
0.48
0.15
0.09 5
0.17
0.43
0.92
0.32
0.69 6
0.34
0.77
0.66
0.8
0.11
0.65
0.01
0.52
0.06 7
0.76
0.27
0.46
0.12
0.58
0.41
0.56 8
0.1
0.57
0.39 9
0.51
0.67
0.07 10
0.03
0.84
0.18
0.13
0.68 11
0.63
0.79 12 13
0.62
0.44
0.47 14
0.81
0.87 15
0.73
0.04 ……
2019/2/23

13. Image single-class assignment
Same as calculating VS correlation values, Using bidirectional associative memories to assign image to single-class
Assignment criteria (L classes being pre-defined)
2019/2/23

14. Classed-based image/key-frame representation
Each image/key-frame is represented by a vector of associative values with category i　regarding domain k
where
Ws, Wv: weights adjusting the weight of SS and VS in generating the associative values
: weights
2019/2/23

15. Data structure for representing image/video
type …
Path1
Path2
Path3 1
image
0.2
0.6
0.7
Path11 2
video
0.3
0.8
Path12
Path22
Path23
k: domain’s number
i:　category’s number
path1: image/key-frames’ location
path2： shots’ location
Path3: videos’ location
ID： image/key-frames’ number
: associative value with category i regarding domain k
2019/2/23

16. Image/key-frame retrieval
For the image categorization regarding single domain
For the image categorization regarding cross-domains
For the image categorization regarding single domain , images are grouped to a class i , when the associative values of these images with class i are larger than a clustering threshold ,
For the image categorization regarding cross-domains, the grouped images are the intersection of those which either belong to the class i in domain k, or belong to the class j in domain l.
2019/2/23

17. Performance evaluation (1)
Influence when varying the number of defined classes
For the precision-recall of class building
2019/2/23

18. Performance evaluation (2)
Influence when varying the resolution of stored pattern image
For the precision-recall of class building while 30 classes were pre-defined
2019/2/23

19. Performance evaluation (3)
Influence when varying the weight values
For the precision-recall of class building when 30 classes were pre-defined
2019/2/23

20. Conclusion & Future work
Method of representing images’ semantics by associative values of images with pre-defined classes is effective for image retrieval
Generating associative values by considering two facts of SS and VS is effective
Bidirectional associative memories is efficient in generating associative values
Evaluating the influence of increasing defined classes on precision-recall
2019/2/23