SWE 423: Multimedia Systems Chapter 4: Graphics and Images (3)

Outline Image Processing Basics Textbook Section 4.3 Image Features Image Segmentation Textbook Section 4.3 Additional Reference: Wasfi Al-Khatib, Y. Francis Day, Arif Ghafoor, and P. Bruce Berra. Semantic modeling and knowledge representation in multimedia databases. IEEE Transactions on Knowledge and Data Engineering, 11(1):64-80, 1999.

Image Processing Image processing involves the analysis of scenes or the reconstruction of models from images representing 2D or 3D objects. Image Analysis Identifying Image Properties (Image Features) Image Segmentation Image Recognition We will look at image processing from a database perspective. Objective: Design of robust image processing and recognition techniques to support semantic modeling, knowledge representation, and querying of images.

Semantic Modeling and Knowledge Representation in Image Databases Feature Extraction. Salient Object Identification. Content-Based Indexing and Retrieval. Query Formulation and Processing.

Multi-Level Abstraction Semantic Modeling And Knowledge Representation Layer Semantic Specification Knowledge Base Semantic Identification Process Object Recognition Layer Object Models Object Recognition Process Feature Extraction Layer Feature Specification Feature Extraction Process Image Data Still Video Frames Multimedia Data

Feature Extraction Layer Image features: Colors, Textures, Shapes, Edges, ...etc. Features are mapped into a multi-dimensional feature space allowing similarity-based retrieval. Features can be classified into two types: Global and Local.

Global Features Generally emphasize coarse-grained pattern matching techniques. Transform the whole image into a functional representation. Finer details within individual parts of the image are ignored. Examples: Color histograms and coherence vectors, Texture, Fast Fourier Transform, Hough Transform, and Eigenvalues. What are some of the example queries?

Color Histogram How many pixels of the image take a specific color In order to control the number of colors, the domain is discretized E.g. consider the value of the two leftmost bits in each color channel (RGB). In this case , the number of different colors is equal to __________ How can we determine whether two images are similar using the color histogram?

Color Coherence Vector Based on the color histogram Each pixel is checked as to whether it is within a sufficiently large one-color environment or not. i.e. in a region related by a path of pixels of the same color If so, the pixel is called coherent, otherwise incoherent For each color j, compute the number of coherent and incoherent pixels (j , j), j = 1, ..., J When comparing two images with color coherence vectors (j , j) and (j , j), j = 1, ..., J, we may use the expression

Texture Texture is a small surface structure Examples include Natural or artificial Regular or irregular Examples include Wood barks Knitting patterns The surface of a sponge

Texture Examples Artificial/periodic Artificial/non-periodic Photographic/pseudo-periodic Photographic/random Photographic/structured Inhomogeneous (non-texture)

Texture Two basic approaches to study texture Structural analysis searches for small basic components and an arrangement rule Statistical analysis describes the texture as a whole based on specific attributes (local gray-level variance, regularity, coarseness, orientation, and contrast. Either done in the spatial domain or the spatial frequency domain

Global Features Advantages: Disadvantages: Simple. Low computational complexity. Disadvantages: Low accuracy

Local Features Images are segmented into a collection of smaller regions, with each region representing a potential object of interest (fine-grained). An object of interest may represent a simple semantic object (e.g. a round object). Choice of features is domain specific: X-ray imaging, GIS, ...etc require spatial features (e.g. shapes [may be calculated through edges] and dimensions.) Paintings, MMR imaging, ...etc may use color features in specific regions of the image.

Edge Detection A given input image E is used to gradually compute a (zero-initialized) output image A. A convolution mask runs across E pixel by pixel and links the entries in the mask at each position that M occupies in E with the gray value of the underlying image dots. The result of the linkage (and the subsequent sum across all products from the mask entry and the gray value of the underlying image pixel) is written to the output image A.

Convolution Convolution is a simple mathematical operation which is fundamental to many common image processing operators. Convolution provides a way of `multiplying together' two arrays of numbers, generally of different sizes, but of the same dimensionality, to produce a third array of numbers of the same dimensionality. This can be used in image processing to implement operators whose output pixel values are simple linear combinations of certain input pixel values. The convolution is performed by sliding the kernel over the image, generally starting at the top left corner, so as to move the kernel through all the positions where the kernel fits entirely within the boundaries of the image.

Convolution Computation If the image E has M rows and N columns, and the kernel K has m rows and n columns, then the size of the output image A will have M - m + 1 rows, and N - n + 1 columns and is given by: Example page 60. http://homepages.inf.ed.ac.uk/rbf/HIPR2/sobel.htm

Similarity Metrics Minkowski Distance Weighted Distance Average Distance Color Histogram Intersection

Prototype Systems QBIC (http://www.hermitagemuseum.org) Uses color, shape, and texture features Allows queries by sketching features and providing color information Chabot (Cypress) Uses color and textual annotation. Improved performance due to textual annotation (Concept Query) KMeD Uses shapes and contours as features. Features are extracted automatically in some cases and manually in other cases.

Demo (Andrew Berman & Linda G. Shapiro ) http://www.cs.washington.edu/research/imagedatabase/demo/seg/ http://www.cs.washington.edu/research/imagedatabase/demo/edge/ http://www.cs.washington.edu/research/imagedatabase/demo/fids/