1. Visual Information Systems Lilian Tang

2. Description of Content – image processing Primitive image properties Primitive image properties Through image processing techniques Through image processing techniques Colour, texture, local shape Colour, texture, local shape The need of combination of these properties into a consistent set of localised properties The need of combination of these properties into a consistent set of localised properties There can be weighting scheme to balance the importance of each type of property. There can be weighting scheme to balance the importance of each type of property. Image features Image features

3. Color One is that the recorded color varies considerably with the orientation of the surface, the viewpoint of the camera, the position of the illumination, the spectrum of the illuminant, and the way the light interacts with the object. One is that the recorded color varies considerably with the orientation of the surface, the viewpoint of the camera, the position of the illumination, the spectrum of the illuminant, and the way the light interacts with the object. Second, the human perception of color is an intricate topic where many attempts have been made to capture perceptual similarity. Second, the human perception of color is an intricate topic where many attempts have been made to capture perceptual similarity.

4. Colour

7. Texture Texture is a phenomenon that is widespread, easy to recognise and hard to define. Texture is a phenomenon that is widespread, easy to recognise and hard to define.

8. Texture Together with colour, texture is a powerful discriminating feature, present almost everywhere in nature. Together with colour, texture is a powerful discriminating feature, present almost everywhere in nature. Like colours, textures are connected with psychological effects. In particular, they emphasize orientations and spatial depth between overlapping object. Like colours, textures are connected with psychological effects. In particular, they emphasize orientations and spatial depth between overlapping object.

9. Traditional Definition of Texture Texture refers to a spatially repeating pattern on a surface that can be sensed visually Texture refers to a spatially repeating pattern on a surface that can be sensed visually In the image, the apparent size, shape, spacing etc, of the texture elements (the texels) do indeed vary In the image, the apparent size, shape, spacing etc, of the texture elements (the texels) do indeed vary Varying distances of the different texels from the camera Varying distances of the different texels from the camera Varying foreshortening of the different texels. Varying foreshortening of the different texels. texture gradients - systematic change in the size and shape of the elements making up a texture texture gradients - systematic change in the size and shape of the elements making up a texture

10.  recover shape from texture

11. Recover Shape From Texture After some mathematical analysis, one can compute expressions for the rate of change of various image texel features, such as area, foreshortening, and density. These texture gradients are functions of the surface shape as well as its slant and tilt with respect to the viewer. After some mathematical analysis, one can compute expressions for the rate of change of various image texel features, such as area, foreshortening, and density. These texture gradients are functions of the surface shape as well as its slant and tilt with respect to the viewer. To recover shape from texture, one can use two-step process: To recover shape from texture, one can use two-step process: 1) measure the texture gradients 1) measure the texture gradients 2) estimate the surface shape, slant, and tilt that would give rise to the measured texture gradients. 2) estimate the surface shape, slant, and tilt that would give rise to the measured texture gradients.

12. Recent Recent Technical Definition of texture Texture is a broad term used in pattern recognition to identify image patches (of any size) that are characterized by differences in brightness. Techniques to extract meaningful texture descriptors from image are many, based on different models and assumptions. An effective representation of textures can be based on statistical and structural properties of brightness patterns.

13. Texture Content Measurement Textures may be described according to their spatial, frequency or perceptual properties. Periodicity, coarseness, preferred direction, degree of complexity are some of the most perceptually salient attributes of a texture. Textures may be described according to their spatial, frequency or perceptual properties. Periodicity, coarseness, preferred direction, degree of complexity are some of the most perceptually salient attributes of a texture. Feature spaces based on these attributes are particularly interesting for image retrieval by texture similarity. Feature spaces based on these attributes are particularly interesting for image retrieval by texture similarity.

14. Space – based models Auto-correlation function A texture can be represented taking into account the spatial size of grey-level primitives. Fine textures have a small size of their grey-level primitives. Coarse textures a large size. Auto-correlation function A texture can be represented taking into account the spatial size of grey-level primitives. Fine textures have a small size of their grey-level primitives. Coarse textures a large size. Co – occurrence matrix A different way of measuring textures is by taking into account the spatial arrangement of grey-level primitives. Co – occurrence matrix A different way of measuring textures is by taking into account the spatial arrangement of grey-level primitives.

15. Calculate Texture energy entropy contrast

17. Integration of primitive properties a separation between color, local geometry, and texture. a separation between color, local geometry, and texture. an integrated view on color, texture, and local geometry is urgently needed as only an integrated view on local properties can provide the means to distinguish among hundreds of thousands different images. an integrated view on color, texture, and local geometry is urgently needed as only an integrated view on local properties can provide the means to distinguish among hundreds of thousands different images. Further research is needed in the design of complete sets of image properties with well- described variant conditions which they are capable of handling. Further research is needed in the design of complete sets of image properties with well- described variant conditions which they are capable of handling.

18. Image features

19. Grouping Data, Global and Accumulating Features, Salient Features, Signs, Shape and Object Features, Description of Structure and Lay-Out Grouping Data, Global and Accumulating Features, Salient Features, Signs, Shape and Object Features, Description of Structure and Lay-Out Also in the description of the image by features, it should be kept in mind that for retrieval a total understanding of the image is rarely needed. Also in the description of the image by features, it should be kept in mind that for retrieval a total understanding of the image is rarely needed. the deeper one goes into the semantics of the pictures, the deeper the understanding of the picture will also have to be the deeper one goes into the semantics of the pictures, the deeper the understanding of the picture will also have to be With segmentation With segmentation no segmentation no segmentation

20. Interpretation And Similarity Measure Semantic features aim at encoding interpretations of the image which may be relevant to the application. Semantic features aim at encoding interpretations of the image which may be relevant to the application. feature set can be explained feature set can be explained derives an unilateral interpretation from the feature set derives an unilateral interpretation from the feature set compares the feature set with the elements in a given data set on the basis of a similarity function compares the feature set with the elements in a given data set on the basis of a similarity function In content-based retrieval, it is useful to push the semantic interpretation of features derived from the image as far as one can. In content-based retrieval, it is useful to push the semantic interpretation of features derived from the image as far as one can.

21. Similarity Measurement A different road to assigning a meaning to an observed feature set, is to compare a pair of observations by a similarity function. – a kind of interpretation A different road to assigning a meaning to an observed feature set, is to compare a pair of observations by a similarity function. – a kind of interpretation And this is the advantage to have content- based retrieval. And this is the advantage to have content- based retrieval.

22. Colour Histogram Based Retrieval This method is to retrieve images from the database that have perceptually similar colour to the input image or input description from the user. This method is to retrieve images from the database that have perceptually similar colour to the input image or input description from the user. The basic idea is to quantize each of the RGB values into m intervals resulting in a total number of m 3 colour combinations (or bins) The basic idea is to quantize each of the RGB values into m intervals resulting in a total number of m 3 colour combinations (or bins) A colour histogram H(I) is then constructed. This colour histogram is a vector {h1, h2, …, hm 3 } where element hx represents the number of pixels in image I falling within bin x. A colour histogram H(I) is then constructed. This colour histogram is a vector {h1, h2, …, hm 3 } where element hx represents the number of pixels in image I falling within bin x.

23. Colour Histogram Based Retrieval The colour histogram becomes the index of this image The colour histogram becomes the index of this image To retrieve image from the database, the user supplies either a sample image or a specification for the system to construct a colour histrogram H(Q). To retrieve image from the database, the user supplies either a sample image or a specification for the system to construct a colour histrogram H(Q). A distance metric is used to measure the similarity between H(Q) and H(I). Where I represents each of the images in the database. And example distance metric is shown as follows: A distance metric is used to measure the similarity between H(Q) and H(I). Where I represents each of the images in the database. And example distance metric is shown as follows: x=m 3 x=m 3 D(Q, I ) =  |q x -i x | D(Q, I ) =  |q x -i x | x=1 x=1 Where q x and i x are the numbers of pixels in the image Q and I, respectively, falling within bin x. Where q x and i x are the numbers of pixels in the image Q and I, respectively, falling within bin x.

24. Colour Histogram Based Retrieval may fail in recognizing images with perceptually similar colours but no common colours. This may be due to a shift in colour values, noise or change in illumination. – measure the similarity may fail in recognizing images with perceptually similar colours but no common colours. This may be due to a shift in colour values, noise or change in illumination. – measure the similarity Not enough for complicated images where spatial position is more important information. Not enough for complicated images where spatial position is more important information. May combine with other methods such as shape and /or texture based retrieval to improve the accuracy of the retrieval. May combine with other methods such as shape and /or texture based retrieval to improve the accuracy of the retrieval.

25. Semantic Similarity knowledge-based type abstraction hierarchies knowledge-based type abstraction hierarchies concept-space concept-space a linguistic description of texture patch visual qualities is given and ordered in a hierarchy of perceptual importance on the basis of extensive psychological experimentation. a linguistic description of texture patch visual qualities is given and ordered in a hierarchy of perceptual importance on the basis of extensive psychological experimentation. A more general concept of similarity is needed for relevance feedback, in which similarity with respect to an ensemble of images is required. A more general concept of similarity is needed for relevance feedback, in which similarity with respect to an ensemble of images is required.

26. Different Levels of Content-base Indexing and Retrieval syntactical level: mainly deal with colour, shape, texture etc. Some used manual annotation to index data syntactical level: mainly deal with colour, shape, texture etc. Some used manual annotation to index data e.g. retrieval system let users to fill forms to provide queries, like location, colour etc categories, like the work done in Berkeley e.g. retrieval system let users to fill forms to provide queries, like location, colour etc categories, like the work done in Berkeley semantic level : semantic level : analyse captions analyse captions purely used text information and didn’t make use of the information inherent in the images purely used text information and didn’t make use of the information inherent in the images complicated algorithm applied on small scale complicated algorithm applied on small scale

27. Multi-level indexing An advantage of image indexing based on multi-level contents rather than solely on low-level features such as texture and colours, is that it would readily provide the basic framework required for "semantic interoperability" when one tries to search through, not only one, but a federation of image collections from different disciplines. An advantage of image indexing based on multi-level contents rather than solely on low-level features such as texture and colours, is that it would readily provide the basic framework required for "semantic interoperability" when one tries to search through, not only one, but a federation of image collections from different disciplines.

28. Learning from Feedback The interacting user brings about many new challenges for the response time of the system. The interacting user brings about many new challenges for the response time of the system. Content-based image retrieval is only scalable to large data sets when the database is able to anticipate what interactive queries will be made. Content-based image retrieval is only scalable to large data sets when the database is able to anticipate what interactive queries will be made. A frequent assumption is that the image set, the features, and the similarity function are known in advance. In a truly interactive session, the assumptions are no longer valid. A frequent assumption is that the image set, the features, and the similarity function are known in advance. In a truly interactive session, the assumptions are no longer valid. A change from static to dynamic indexing is required. (Arnold 2000) A change from static to dynamic indexing is required. (Arnold 2000)

29. An integrated issue It will demand its own view of things as it is our belief that content-based retrieval in the end will not be part of the field of computer vision alone. The man-machine interface, domain knowledge, and database technology each will have their impact on the product. It will demand its own view of things as it is our belief that content-based retrieval in the end will not be part of the field of computer vision alone. The man-machine interface, domain knowledge, and database technology each will have their impact on the product.

30. Arnold 2000

31. Summary The heritage of computer vision. The heritage of computer vision. The influence on computer vision. The influence on computer vision. Deal with large data sets. Deal with large data sets. the absence of a general method for strong segmentation. the absence of a general method for strong segmentation. has revitalized interest in color image processing. has revitalized interest in color image processing. attention for invariance has been revitalized attention for invariance has been revitalized Similarity and learning. Similarity and learning. Interaction. Interaction. The need for databases. The need for databases. The problem of evaluation. The problem of evaluation. The semantic gap and other sources. The semantic gap and other sources.