Visual Information Systems Image Content

Visual cues to recover 3-D information There are number of cues available in the visual stimulus There are number of cues available in the visual stimulus Motion Motion Binocular stereopsis Binocular stereopsis Texture Texture Shading Shading Contour Contour These cues are therefore often used as features to represent image content These cues are therefore often used as features to represent image content

Description of Content – image processing Primitive image properties Primitive image properties Through image processing techniques Through image processing techniques Colour, texture, local shape Colour, texture, local shape The need of combination of these properties into a consistent set of localised properties The need of combination of these properties into a consistent set of localised properties There can be weighting scheme to balance the importance of each type of property. There can be weighting scheme to balance the importance of each type of property. Image features Image features

Integration of primitive properties an integrated view on colour, texture, and local geometry is needed as only an integrated view on local properties can provide the means to distinguish among hundreds of thousands different images. an integrated view on colour, texture, and local geometry is needed as only an integrated view on local properties can provide the means to distinguish among hundreds of thousands different images. Further research is needed in the design of complete sets of image properties with well- described variant conditions which they are capable of handling. Further research is needed in the design of complete sets of image properties with well- described variant conditions which they are capable of handling. This is essential as the balance between stability against variations and retained discriminatory power determines the effectiveness of a property. This is essential as the balance between stability against variations and retained discriminatory power determines the effectiveness of a property.

Image features/properties What is it? What is it? How to detect it? How to detect it? How to measure it? How to measure it? How to use it? How to use it?

Colour The recorded colour varies considerably with the orientation of the surface, the viewpoint of the camera, the position of the illumination, the spectrum of the illuminant, and the way the light interacts with the object. The recorded colour varies considerably with the orientation of the surface, the viewpoint of the camera, the position of the illumination, the spectrum of the illuminant, and the way the light interacts with the object. The human perception of colour is an intricate topic where many attempts have been made to capture perceptual similarity. The human perception of colour is an intricate topic where many attempts have been made to capture perceptual similarity.

Colour Histogram Based Retrieval This method is to retrieve images from the database that have perceptually similar colour to the input image or input description from the user. This method is to retrieve images from the database that have perceptually similar colour to the input image or input description from the user. The basic idea is to quantize each of the RGB values into m intervals resulting in a total number of m 3 colour combinations (or bins) The basic idea is to quantize each of the RGB values into m intervals resulting in a total number of m 3 colour combinations (or bins) A colour histogram H(I) is then constructed. This colour histogram is a vector {h1, h2, …, hm 3 } where element hx represents the number of pixels in image I falling within bin x. A colour histogram H(I) is then constructed. This colour histogram is a vector {h1, h2, …, hm 3 } where element hx represents the number of pixels in image I falling within bin x.

Colour Histogram Based Retrieval The colour histogram becomes the index of this image The colour histogram becomes the index of this image To retrieve image from the database, the user supplies either a sample image or a specification for the system to construct a colour histrogram H(Q). To retrieve image from the database, the user supplies either a sample image or a specification for the system to construct a colour histrogram H(Q). A distance metric is used to measure the similarity between H(Q) and H(I). Where I represents each of the images in the database. And example distance metric is shown as follows: A distance metric is used to measure the similarity between H(Q) and H(I). Where I represents each of the images in the database. And example distance metric is shown as follows: x=m 3 x=m 3 D(Q, I ) =  |q x -i x | or use Euclidean distance measure D(Q, I ) =  |q x -i x | or use Euclidean distance measure x=1 x=1 Where q x and i x are the numbers of pixels in the image Q and I, respectively, falling within bin x. Where q x and i x are the numbers of pixels in the image Q and I, respectively, falling within bin x.

Colour Histogram Based Retrieval may fail in recognizing images with perceptually similar colours but no common colours. This may be due to a shift in colour values, noise or change in illumination. – measure the similarity may fail in recognizing images with perceptually similar colours but no common colours. This may be due to a shift in colour values, noise or change in illumination. – measure the similarity Not enough for complicated images where spatial position is more important information. Not enough for complicated images where spatial position is more important information. May combine with other methods such as shape and /or texture based retrieval to improve the accuracy of the retrieval. May combine with other methods such as shape and /or texture based retrieval to improve the accuracy of the retrieval.

Texture Texture is a phenomenon that is widespread, easy to recognise and hard to define. Texture is a phenomenon that is widespread, easy to recognise and hard to define.

Texture Together with colour, texture is a powerful discriminating feature, present almost everywhere in nature. Together with colour, texture is a powerful discriminating feature, present almost everywhere in nature. Like colours, textures are connected with psychological effects. In particular, they emphasize orientations and spatial depth between overlapping object. Like colours, textures are connected with psychological effects. In particular, they emphasize orientations and spatial depth between overlapping object.

Texture There are three standard problems to do with texture: There are three standard problems to do with texture: Texture segmentation is the problem of breaking an image into components within which the texture is constant. Texture segmentation involves both representing a texture, and determining the basis on which segment boundaries are to be determined. Texture segmentation is the problem of breaking an image into components within which the texture is constant. Texture segmentation involves both representing a texture, and determining the basis on which segment boundaries are to be determined. Texture synthesis seeks to construct large regions of texture from small example images. Texture synthesis seeks to construct large regions of texture from small example images. Shape from texture involves recovering surface orientation or surface shape from image texture. Shape from texture involves recovering surface orientation or surface shape from image texture.

Traditional Definition of Texture Texture refers to a spatially repeating pattern on a surface that can be sensed visually Texture refers to a spatially repeating pattern on a surface that can be sensed visually In the image, the apparent size, shape, spacing etc, of the texture elements (the texels) do indeed vary In the image, the apparent size, shape, spacing etc, of the texture elements (the texels) do indeed vary Varying distances of the different texels from the camera Varying distances of the different texels from the camera Varying foreshortening of the different texels. Varying foreshortening of the different texels. texture gradients - systematic change in the size and shape of the elements making up a texture texture gradients - systematic change in the size and shape of the elements making up a texture

Recover Shape From Texture After some mathematical analysis, one can compute expressions for the rate of change of various image texel features, such as area, foreshortening, and density. These texture gradients are functions of the surface shape as well as its slant and tilt with respect to the viewer. After some mathematical analysis, one can compute expressions for the rate of change of various image texel features, such as area, foreshortening, and density. These texture gradients are functions of the surface shape as well as its slant and tilt with respect to the viewer. To recover shape from texture, one can use two-step process: To recover shape from texture, one can use two-step process: 1) measure the texture gradients 1) measure the texture gradients 2) estimate the surface shape, slant, and tilt that would give rise to the measured texture gradients. 2) estimate the surface shape, slant, and tilt that would give rise to the measured texture gradients.

Recent Technical Definition of texture:- Texture is a broad term used in pattern recognition to identify image patches (of any size) that are characterized by differences in brightness.

Techniques to extract meaningful texture descriptors from image are many, based on different models and assumptions. An effective representation of textures can be based on statistical and structural properties of brightness patterns.

Texture Content Measurement Textures may be described according to their spatial, frequency or perceptual properties. Periodicity, coarseness, preferred direction, degree of complexity are some of the most perceptually salient attributes of a texture. Textures may be described according to their spatial, frequency or perceptual properties. Periodicity, coarseness, preferred direction, degree of complexity are some of the most perceptually salient attributes of a texture. Feature spaces based on these attributes are particularly interesting for image retrieval by texture similarity. Feature spaces based on these attributes are particularly interesting for image retrieval by texture similarity.

Structural Texture Representations Require Require texture primitive - texel texture primitive - texel placement rule placement rule Ideal for regular - man-made - textures Ideal for regular - man-made - textures

Statistical Descriptions Better suited to pseudorandom, natural textures Better suited to pseudorandom, natural textures First Order statistics First Order statistics Second order statistics Second order statistics

First Order Statistics Statistical descriptions of grey level distribution Statistical descriptions of grey level distribution Mean grey value Mean grey value Deviation of grey values Deviation of grey values Coefficient of variation Coefficient of variation etc. etc.

Second Order Statistics Measures involving multiple pixels Measures involving multiple pixels Joint difference histogram Joint difference histogram histogram of differences between adjacent pixels histogram of differences between adjacent pixels Co-Occurrence matrices Co-Occurrence matrices measure frequency of specific pairs of grey values measure frequency of specific pairs of grey values

Space – based models Auto-correlation function A texture can be represented taking into account the spatial size of grey-level primitives. Fine textures have a small size of their grey-level primitives. Coarse textures a large size. Auto-correlation function A texture can be represented taking into account the spatial size of grey-level primitives. Fine textures have a small size of their grey-level primitives. Coarse textures a large size. Co – occurrence matrix A different way of measuring textures is by taking into account the spatial arrangement of grey-level primitives. Co – occurrence matrix A different way of measuring textures is by taking into account the spatial arrangement of grey-level primitives.

energy entropy contrast

Edge Frequency Density of microedges is characteristic of texture Density of microedges is characteristic of texture Apply an edge detector Apply an edge detector Sobel is suitable Sobel is suitable Threshold result Threshold result Compute density of edge elements Compute density of edge elements

Image features

Grouping Data, Global and Accumulating Features, Salient Features, Signs, Shape and Object Features, Description of Structure and Lay-Out Grouping Data, Global and Accumulating Features, Salient Features, Signs, Shape and Object Features, Description of Structure and Lay-Out Also in the description of the image by features, it should be kept in mind that for retrieval a total understanding of the image is rarely needed. Also in the description of the image by features, it should be kept in mind that for retrieval a total understanding of the image is rarely needed. The deeper one goes into the semantics of the pictures, the deeper the understanding of the picture will also have to be The deeper one goes into the semantics of the pictures, the deeper the understanding of the picture will also have to be With segmentation With segmentation No segmentation No segmentation

Interpretation and Similarity Measure Semantic features aim at encoding interpretations of the image which may be relevant to the application. Semantic features aim at encoding interpretations of the image which may be relevant to the application. feature set can be explained feature set can be explained derives an unilateral interpretation from the feature set derives an unilateral interpretation from the feature set compares the feature set with the elements in a given data set on the basis of a similarity function compares the feature set with the elements in a given data set on the basis of a similarity function

Similarity Measurement A different road to assigning a meaning to an observed feature set, is to compare a pair of observations by a similarity function. – a kind of interpretation A different road to assigning a meaning to an observed feature set, is to compare a pair of observations by a similarity function. – a kind of interpretation And this is the advantage to have content- based retrieval. And this is the advantage to have content- based retrieval.

Semantic Similarity knowledge-based type abstraction hierarchies knowledge-based type abstraction hierarchies concept-space concept-space a linguistic description of texture patch visual qualities is given and ordered in a hierarchy of perceptual importance on the basis of extensive psychological experimentation. a linguistic description of texture patch visual qualities is given and ordered in a hierarchy of perceptual importance on the basis of extensive psychological experimentation.

Learning from Feedback The interacting user brings about many new challenges for the response time of the system. The interacting user brings about many new challenges for the response time of the system. Content-based image retrieval is only scalable to large data sets when the database is able to anticipate what interactive queries will be made. Content-based image retrieval is only scalable to large data sets when the database is able to anticipate what interactive queries will be made. A frequent assumption is that the image set, the features, and the similarity function are known in advance. In a truly interactive session, the assumptions are no longer valid. A frequent assumption is that the image set, the features, and the similarity function are known in advance. In a truly interactive session, the assumptions are no longer valid. A change from static to dynamic indexing is required. (Arnold 2000) A change from static to dynamic indexing is required. (Arnold 2000)

An integrated issue Content-based retrieval in the end will not be part of the field of computer vision alone. The man-machine interface, domain knowledge, and database technology each will have their impact on the product. Content-based retrieval in the end will not be part of the field of computer vision alone. The man-machine interface, domain knowledge, and database technology each will have their impact on the product.