Topic creation for medical image retrieval benchmarks ImageCLEF/MUSCLE workshop, Alicante, Henning Müller, Bill Hersh

2 ©2006 Hôpitaux Universitaires de Genève Overview Image retrieval benchmarking and applicationsImage retrieval benchmarking and applications ComponentsComponents Medical image retrievalMedical image retrieval Finding out more on information needsFinding out more on information needs Analysis of the content of our datasetAnalysis of the content of our dataset Surveys among professional usersSurveys among professional users Log file analysis (foundation health on the net)Log file analysis (foundation health on the net) ExamplesExamples ConclusionsConclusions

3 ©2006 Hôpitaux Universitaires de Genève Image retrieval and evaluation Retrieval vs. classificationRetrieval vs. classification Nothing is know on a retrieval datasetNothing is know on a retrieval dataset In other domains standard datasets have existed for a long timeIn other domains standard datasets have existed for a long time Text retrieval, segmentation, character recognition, …Text retrieval, segmentation, character recognition, … Image retrieval starts getting betterImage retrieval starts getting better BenchathlonBenchathlon TRECVIDTRECVID ImageCLEFImageCLEF ImageEval, …ImageEval, …

4 ©2006 Hôpitaux Universitaires de Genève Components of a benchmark A datasetA dataset Large (! Problems are different !)Large (! Problems are different !) Realistic with respect to a certain user modelRealistic with respect to a certain user model Annotation, etc.Annotation, etc. Query topics based on real information needsQuery topics based on real information needs Participants for comparisonParticipants for comparison Ground truth/Relevance judgmentsGround truth/Relevance judgments Performance measuresPerformance measures WorkshopWorkshop Foster discussions, not a pure competitionFoster discussions, not a pure competition

5 ©2006 Hôpitaux Universitaires de Genève Medical image retrieval Research domainResearch domain Users are often technophobeUsers are often technophobe Frequently proposed as important (potential) but never really used in practiceFrequently proposed as important (potential) but never really used in practice A single study on diagnostic useA single study on diagnostic use Most users work with Google but do not know anything about visual retrievalMost users work with Google but do not know anything about visual retrieval Problems and possibilitiesProblems and possibilities Use on varied dataset vs. Diagnostic aid (very specific databases)Use on varied dataset vs. Diagnostic aid (very specific databases)

6 ©2006 Hôpitaux Universitaires de Genève Motivation Find out more on the behavior of medical professionals regarding the use of imagesFind out more on the behavior of medical professionals regarding the use of images How is searched for images?How is searched for images? What can be useful in the future?What can be useful in the future? Educate them on the techniques available and their possibilitiesEducate them on the techniques available and their possibilities Stimulate creativity to learn about potentially good applicationsStimulate creativity to learn about potentially good applications BrainstormingBrainstorming Goal is multimodal image retrieval (visual included)Goal is multimodal image retrieval (visual included) For ImageCLEFmedFor ImageCLEFmed

7 ©2006 Hôpitaux Universitaires de Genève ImageCLEF 2004 Query topics were images, onlyQuery topics were images, only Radiologist familiar with the database choose themRadiologist familiar with the database choose them Represent the database well with its variabilityRepresent the database well with its variability Text could be used for subsequent stepsText could be used for subsequent steps Goal was to retrieve images similar/same in anatomic region, modality, and viewGoal was to retrieve images similar/same in anatomic region, modality, and view Well defined task... but is this realistic?Well defined task... but is this realistic? User model MDUser model MD Would they search with an image only?Would they search with an image only? How to get the image?How to get the image?

8 ©2006 Hôpitaux Universitaires de Genève Surveys among medical professionals In Portland and GenevaIn Portland and Geneva Separated by functionSeparated by function LibrarianLibrarian StudentStudent LecturerLecturer ResearcherResearcher ClinicianClinician Get typical search tasks as examplesGet typical search tasks as examples From various departmentsFrom various departments Qualitative, not too time consumingQualitative, not too time consuming

9 ©2006 Hôpitaux Universitaires de Genève Questions at the survey   What kind of tasks do you perform in your daily work where images are useful for you?   For each of these tasks, can you give us an example of what kind of image you are searching for?   For each of these tasks, where do you search for the images? (Ordered by preference)   When you search for images, how do you search for them?   When you find an image, how do you decide whether one or another corresponds to your needs?   What search tools or functions would be useful for you to search for images in addition to what is currently used?

10 ©2006 Hôpitaux Universitaires de Genève Some results Search tasks vary strongly between functionsSearch tasks vary strongly between functions Clinicians often do not have much choiceClinicians often do not have much choice Access per patient and by patient idAccess per patient and by patient id Several people did not know about visual retrievalSeveral people did not know about visual retrieval Retrieval for pathology was regarded as most importantRetrieval for pathology was regarded as most important And currently not possibleAnd currently not possible Retrieval of similar cases was proposed as very useful several timesRetrieval of similar cases was proposed as very useful several times

11 ©2006 Hôpitaux Universitaires de Genève Log file analysis of a medical media search Health On the Net ( On the Net ( 35‘000 query terms of a one year query log35‘000 query terms of a one year query log HONmedia search for medial images and videosHONmedia search for medial images and videos Spelling errorsSpelling errors Several languagesSeveral languages Calculate frequencies of term combinationsCalculate frequencies of term combinations Removal of media types (images, photos, videos, …) from the queriesRemoval of media types (images, photos, videos, …) from the queries Removal of frequent spelling errorsRemoval of frequent spelling errors Change of word order (alphabetic)Change of word order (alphabetic)

12 ©2006 Hôpitaux Universitaires de Genève Some results Half of the queries are unique!!Half of the queries are unique!! Almost the majority of queries contains one wordAlmost the majority of queries contains one word Queries are most often not specific at allQueries are most often not specific at all Risk to have thousands of results!!Risk to have thousands of results!! HeartHeart LungLung Images/videosImages/videos People do not only search for health subjectsPeople do not only search for health subjects Few very specific questionsFew very specific questions But these were very specific!But these were very specific!

13 ©2006 Hôpitaux Universitaires de Genève Other sources … Content of the data base needs to be taken into account to have varied queriesContent of the data base needs to be taken into account to have varied queries Frequent causes of death are most important (CDC)Frequent causes of death are most important (CDC) Develop variety along four axesDevelop variety along four axes ModalityModality Anatomic regionAnatomic region PathologyPathology Visual observationVisual observation

14 ©2006 Hôpitaux Universitaires de Genève... and constraints Number of relevant items needs to be limitedNumber of relevant items needs to be limited Otherwise we would miss many relevantOtherwise we would miss many relevant There should be at least a few relevant itemsThere should be at least a few relevant items How to choose images for the queriesHow to choose images for the queries From collection, modified, from the web?From collection, modified, from the web? We would like visual, mixed and semantic queriesWe would like visual, mixed and semantic queries Satisfy all participantsSatisfy all participants Create candidates and then reduce numberCreate candidates and then reduce number Create unambiguous topics!Create unambiguous topics! A negative description for judges can helpA negative description for judges can help

15 ©2006 Hôpitaux Universitaires de Genève Examples 2005 Show me x-ray images with fractures of the femur. Zeige mir Röntgenbilder mit Brüchen des Oberschenkelknochens. Montre-moi des fractures du fémur. Show me chest CT images with emphysema. Zeige mir Lungen CTs mit einem Emphysem. Montre-moi des CTs pulmonaires avec un emphysème. Show me any photograph showing malignant melanoma. Zeige mir Bilder bösartiger Melanome. Montre-moi des images de mélanomes malignes.

16 ©2006 Hôpitaux Universitaires de Genève Example Show me x-ray images of bone cysts. Zeige mir Röntgenbilder von Knochenzysten. Montre-moi des radiographies de kystes d'os.

17 ©2006 Hôpitaux Universitaires de Genève Example 2006 (2) 1.4 Show me x-ray images of a tibia with a fracture. Zeige mir Röntgenbilder einer gebrochenen Tibia. Montre-moi des radiographies du tibia avec fracture.

18 ©2006 Hôpitaux Universitaires de Genève Conclusions Topic creation is extremely import for benchmarksTopic creation is extremely import for benchmarks Need to be useful for user model, not purely academicNeed to be useful for user model, not purely academic Several sources can be used even if no real use of system is availableSeveral sources can be used even if no real use of system is available Discussions with professionals can bring up many good ideas (and educate your users)Discussions with professionals can bring up many good ideas (and educate your users) A development in several steps helps to correspond to all constraintsA development in several steps helps to correspond to all constraints Define constraints in advanceDefine constraints in advance Start with a larger number and then reduceStart with a larger number and then reduce But: robustnessBut: robustness

19 ©2006 Hôpitaux Universitaires de Genève Questions?