Visiwords John Tait Chief Scientific Officer

Warning A few half formed ideas from the world of image and video indexing which may be of interest to MT people Not original ideas (apart from I think the link) In fact a line of work which derives originally from MT

The nub Unsupervised Clustering of bundles of features –Colour, texture from image segments –Words, phrases from sentences or paragraphs ? Associate these bundles with “translations” by supervised machine learning –Categorised images –Parallel texts

Origins “Matching Words and Pictures”: Barnard, Duygulu, Forsyth, de Freitas, Blei, Jordan. Journal of Machine Learning Research 3 (2003) “Image Classification Using Hybrid Neural Networks” Tsai, McGarry and Tait. Proceedings of the 26 th ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2003), Toronto, July, pp

More or less general clusters General Concepts Specific Concepts

Visiwords Derived from Visiterms –These feature cluster nodes –A notion of an area of the “semantic field” –Remember these are colour, texture etc. for an area of an image …. No relation to language … or at least a very deep one

Matching L1 General Concepts L1 Specific Concepts L2 General Concepts L2 Specific Concepts Examples

Fast Forward to 2009 Better statistical models tuned to the data Much Bigger vocabularies of words categories … and lots of other advances

A question Is there anything like this current MT research ?

Concluding remarks I’m surprised this worked at all –Why should image data be coherent and cohesive ? But text is !!! Is this a better way to deal with unknown and changing vocabulary

Some other references A Correlation Approach for Automatic Image Annotation Hardoon, D., Saunders, C., Szedmak, S. and Shawe-Taylor, J. (2006) A Correlation Approach for Automatic Image Annotation. In: Second International Conference on Advanced Data Mining and Applications, ADMA 2006, August, Xi'an, China. Kucuktunc, O., Sevil, S. G., Tosun, A. B., Zitouni, H., Duygulu, P., and Can, F Tag Suggestr: Automatic Photo Tag Expansion Using Visual Information for Photo Sharing Websites. In Proceedings of the 3rd international Conference on Semantic and Digital Media Technologies: Semantic Multimedia (Koblenz, Germany, December , 2008). D. Duke, L. Hardman, A. Hauptmann, D. Paulus, and S. Staab, Eds. Lecture Notes In Computer Science, vol Springer-Verlag, Berlin, Heidelberg, DOI=