1. Multimedia Information Retrieval It is of the highest importance, in the art of detection, to be able to recognize out of a number of facts which are incidental and which are vital... -Sherlock Holmes Presented by Dipti Vaidya

2. Overview of the Presentation Overview of MultiMedia Information Retrieval Research Issues & Approaches Case Study of 2 of these Approaches: - Content Based Approximate Picture Retrieval -Knowledge Based Approach for Retrieving Images by Content What next ?

3. Multimedia Information Retrieval Types of Associated Information -Content-independent metadata (CIM) Format,author’s name,date -Content-dependent metadata (CDepM) Low level features concerned with perceptual facts eg:color,texture,shape,spatial relationship -Content-descriptive metadata (CDesM) High level content semantics eg: good weather

4. New Generation MMIR Retrieval not only by concepts but also by perception of visual content Objective measurements of visual contents and appropriate similarity models Automatically extract features from raw data by image processing, speech recognition,pattern recognition and other computer vision techniques

5. MMIR

6. Image Retrieval By perceptual features -for each image in database,a set of features are computed -to query the image database, through visual examples authored by user or extracted by image samples -select features and choose a similarity measure -Compute similarity degree, ranking and relevance feedback

7. Image retrieval

9. Research Issues Extraction of perceptual feature (CDepM) -Color,Texture,shape:Because of perception subjectivity,there doesn’t exist a single best presentation for a given feature. -Segmentation Comparison of existing methods:

10. Comparision of Existing techniques Histogram Matching : big in size,complicating database creation,sensitive to brightness creation,lacks spatial information,difficult to create partial match Texture Mapping : not applicable to non textured regions,hard to figure out what model to use for a given application Region based mapping: results can go ugly when segmentation is not done properly

11. Research Issues To make CBIR truly scalable to large size image collections, efficient multidimensional indexing techniques needs to be explored Retrieval Speed Multi-Dimensional Indexing techniques: - Bucketing Algorithm - K-d tree - K-D-B tree - R- tree and it’s variants R+ tree and R*- tree Problems: These techniques cluster data based on minimizing the number of disk access per data retrieval, but do not consider semantic difference of image of image features; thus no global conceptual view of image clustering can be provides eg: LARGE, NEARBY tumor ????

12. MMIR Systems QBIC Visual Seek and WebSEEK MARS Photobook

13. Content-Based Approximate Picture Retrieval -A Prasad Sistla, Clement Yu at UIC

14. Problem Formulation The objective is to find the pictures stored in the system that are very similar to the user’s description A similarity measure is used to find the degree by which two descriptions match

15. Solution Approach There is some metadata associated with each picture in our database which describes the content of each picture Metadata contains information about the objects in the picture,their properties and the relationships among them A knowledge base containing a thesaurus and certain rules is used for some deductions

16. Representation of Pictures A picture can be represented by an E-R graph as follows: Entities: objects identified in a picture Relationships: associations between the identified objects Attributes:properties (color,size etc) that qualify of characterize objects

17. Example Example: TREE Right of RIVER Over Bridge Left Of HILL short green blue tall

18. Relationship Classification Action -Mutual eg: handshake -Directional eg: moving Spatial -Mutual eg: adjacent -Directional eg: above Subject Entity Object Entity

19. User Interface Queries are entered by the user via the interactive interface Identify all the objects Identify various attributes of the object eg:color, size Choose the appropriate relationship from the list provided Visual language also provided to specify part of a query

20. Mapping Queries to Formal Language Expressions Using UniSQL/X Example: “ Find the picture(s) of president Clinton” SELECT P.pid FROM Picture P WHERE P.Entities[X] And X.Cname =‘president’ and X.Pname=‘Clinton’ Where Cname = Common Name and Pname = Proper Name Plan to write 2 more query examples…..

21. RETRIEVAL Two pictures are similar if both the corresponding entities and the corresponding relationships are similar Similarity of 2 Entities: 1.The name of the entities are the same, synonyms or in an IS-A relationship hierarchy 2.The attribute values of the 2 entities do not conflict

22. Similarity of 2 Entites Let e be an user specifies entity and E be an entity stored in the system e = {n,a1,a2,…ak} E = {N,A1,A2,…,AK} Where n and N are the name of the entities e and E respectively and ai and Ai are the ith attribute values of the corresponding entities The similarity of entities can now be defined as: k Sim (e,E) = ½ (Sim (n,N) + 1/k  Sim (ai,Ai)) i=1

23. Similarity of 2 Entity names 1.The 2 names are same => Sim(n,N) = 1 2.The 2 names are synonyms => Sim(n,N) >0 3. The 2 names are antonyms => Sim(n,N)= -inf 4. One of the names is NULL or both are NULL=> Sim(n,N) = 0

24. Similarity of 2 Attributes We say that an attribute is neighborly if there is a closeness predicate that determines for any 2 values of the attribute whether the 2 values are close or not. For Eg: The “age” attribute is neighborly it can take one of the values- “very young”, “young”,”middle age”,”old” and “very old” Two values are considered “close” if they occur next to each other in the list

25. Similarity of 2 neighborly attributes: Sim (a,A) = w if a = A Sim (a,A) = cw if a or A are close Sim (a,A) = -inf if a and A are not close Here, w is determined using the inverse document frequency method & c is a positive constant less than 1

26. Similarity of 2 Relationships Let r be the relationship of some entities which are specified in the user’s description and R be another relationship of some entities which are given in a picture stored in the system r = {n,(e1,t1),(e2,t2),…(em,tm)} R= {N,(E1,T1),(E2,T2)…..(Em,Tm) T = type (O-Object, S-Subject) m Sim (r,R) = 1/3 (Sim (n,N) + 1/m  Sim(ei,Ei) +1/m  Sim (ti,Ti)) i = 1 i =1 Sim(ti,Ti) = 1 if ti = Ti = 0 if one is type N and other is not = - inf if one is type S and other is type O

27. A sample Retrieval E-R Diag (Query) Sail Boat River Mountain In by Tree Fisherman Sailor white small still ¦ 5 Black,white medium moving bluemoving Ll,ml,rl blacksmallstill rl green still Lu,mu,ru still largebrown 1 2 4 6

28. A Sample Retrieval E-R Diag (DB-pic1) Rm,ru green still Tree Right of River Over Bridge mm 3 moving blue 1 Lm,mm,rm small stationary brown Left Of Above Mountain sky brown Lm,mm stationary 4 blue stationary Lu,um,ru 5 2

29. A Sample Retrieval E-R Diag (DB-pic2) Sail boat in river Behind tree behind beach Mountain Lm,mm,rm 25 3 stationary green Lu,mu,ru still Very large brown 3d-irregular 5 On person In Person white 6 ll black moving Ll,ml,rl black stationary 7 blue 1 Ll,ml,rl 2

30. Research Issues Not very feasible for large databases Coding of the descriptions of the pictures is a labor intensive task Handling type matches eg:clouds “in” the sky or “cloudy” sky Deduction of Spatial Relationships

31. A Knowledge_Based Approach for Retrieving Images by Content Chih-Cheng Hsu, Wesley Chu -UCLA

32. Overview A Knowledge-based spatial image model (KSIM) which supports queries with semantic and similar-to predicates Interested objects in the images are represented by contours segmented from images Image content are these object contours using domain specific knowledge These image features are classified using MDISC and represented by TAH for knowledge-based query processing

33. KSIM A three-layered model is used to integrate the image representations and image features together with image content interpretation knowledge - The Representation Layer (RL) - The Semantic Layer (SL) - The knowledge Layer (KL)

34. Image Representation Raw images are stored in RL Image objects are represented by contours which can be segmented manually or semi-automatically in the RL Difficulty: automated segmentation of these objects is still not achieved which leads to the deployment of this technique.

35. Semantic Layer The shape model and spatial relationship model in the SL are used to extract image features from the contours Example: Object feature - Conceptual terms Tumor.size small,medium,large Tumor.roundness circular,non-circular Lateral_ventricle. Symmetric, upper_protrusion_pressed_right L_R_Symmetry upper_protrusion_pressed_left

36. Segmentation Layer Spatial relationship table: Spatial Relationship Rep.features defined semantic term SR(t,b) (xc,yc,ia) slightly occupied, extremely occupied SR(t,l) (oc,dc,xc,yc) nearby,far_away …. …..…..

37. Mapping Queries to Formal Language Expression using CoBase Example Query: “find large tumor nearby the lateral ventricle” Select patientWithImage(patient: t.patient,image:t.image) From Tumors t, Lateral_ventricle l Where t NEARBY l and t.size IS ‘large’

38. Query Interpretation via TAH The concept in the TAH node is represented as the value range of the features The TAH nodes can be labels with the conceptual term (eg. large, small) to represent specific knowledge There is a TAH directory that stores such information as object names,set of features, spatial relationships,user type,purpose of TAH,etc Based on this information, the system selects and retrieves the appropriate TAH for processing the query

39. TAH Example:

40. Query Processing The query analysis and feature selection phase The knowledge based content matching phase Query relaxation phase

41. Flow diag of query processing Query Query Processing Satisfactory answers Relaxation manager TAHs, user model Query modification Post Processing

42. A Sample Retrieval Example User: brain surgeon mandatory matched objects:Lesion and Brain optional matched objects: lateral ventricle and frontal lobe relaxation order: SR(l,lv) and SR(l,f) are more important than SR(l,b) in order.. brain SR(l,b) Lesion SR(l,f) SR(l,lv) Frontal Lobe Lateral Ventricle 1 1 2

43. Research Issues Use XML Metadata for describing images and the use of this data for automatic generation of the web pages.