DOG I : an Annotation System for Images of Dog Breeds Antonis Dimas Pyrros Koletsis Euripides Petrakis Intelligent Systems Laboratory Technical University of Crete (TUC) Chania, Crete, Greece

Image Annotation  The task of assigning a name or description to an unknown image  Manual: good quality, but slow, subjective  Automatic: classification problem, relies on associating image analysis features with high level concepts  Difficult to handle all image types  Semantic gap: map features to classes 8/17/2015ICIAR 2012, Aveiro, Portugal2

DOG I :  An automatic image annotation system for images of dog breeds  40 classes (dog breeds)  Descriptions: information in an ontology  Class names, properties, features, textual descriptions (from WordNet, Wikipedia)  Annotations in MPEG7 8/17/2015ICIAR 2012, Aveiro, Portugal3

DOG I : System 8/17/2015 ICIAR 2012, Aveiro, Portugal 4 Graphical User Interface (GUI) Feature Extraction Ontology Mapping Image Annotation DOG I Ontology Load Image Select ROI Annotation Method MPEG7 features Color + Texture features Images: 12-dim vectors 40 classes 9 instances/class Class hierarchy Class properties Image Retrieval Select Annotation Mathod Store Annotation in Exif header

Select ROI 8/17/2015ICIAR 2012, Aveiro, Portugal5

Image Content Analysis  Images of dog breeds are mainly characterized by the spatial distribution of color intensities  A 12-dimension feature vector of Color, Texture, Hybrid feature from LIRE Library  Features are normalized in [0,1]  Not all features are equally important 8/17/2015ICIAR 2012, Aveiro, Portugal6

Ontology  40 classes of dog breed organized in IS_A hierarchy  E.g., Dog  Working Group  Saint Bernard  Three separate hierarchies for text, features and visual descriptions  9 instances per class: raw images + a 12- dim feature vector for each image in class 8/17/2015ICIAR 2012, Aveiro, Portugal7

DOG I Ontology 8/17/2015ICIAR 2012, Aveiro, Portugal8

Image Annotation  The unknown image Q is compared with each one of the 360 images in the ontology  D(Q,I) = Σ i w i d i (Q,I)  Results are ranked by similarity with Q  Weights w i are computed by decision trees  Training set of 3,474 image pairs  8/17/2015ICIAR 2012, Aveiro, Portugal9

Annotation Method  Best Match: Select class of most similar instance  Max Occurrence: Select class with more instances in the first 20 answers  Average Retrieval Rank: Select class with instances ranked higher in the first 20 answers  Max Similarity: Select class whose instancing sum-up to max similar score 8/17/2015ICIAR 2012, Aveiro, Portugal10

Example Image 8/17/2015ICIAR 2012, Aveiro, Portugal11

Annotation Result 8/17/2015ICIAR 2012, Aveiro, Portugal12

EXIF Metadata  Descriptive information embedded inside an image  The metadata captured by your camera is called EXIF data..  DOGI stores annotation info with the pictures in the EXIF  Can be useful for image archiving and later retrievals 8/17/2015ICIAR 2012, Aveiro, Portugal13

Annotation in MPEG7 8/17/2015ICIAR 2012, Aveiro, Portugal14

Evaluation  Average annotation accuracy over 40 queries 8/17/2015ICIAR 2012, Aveiro, Portugal15 Annotation result Max Similarity AVRMax Occurrence Best Match Ranked 1 st 72.5%62.5%65%50% Ranked 2 nd 17.5%22.5%15%10% Ranked 3 rd 5%10% Overall95%92,5%90%

Conclusions-Future Work  DOG Ι : An automatic annotation system for dog breeds with good performance  Useful as a tool for many application  Annotation accuracy improves for less categories  Experimenting with more and animal species images categories  More elaborate image classification methods 8/17/2015ICIAR 2012, Aveiro, Portugal16

THANK YOU !! 8/17/2015ICIAR 2012, Aveiro, Portugal17