1. UMC – HCI seminar series 1 Human Computer Interaction Query by Sketch Chi-Ren Shyu Department of Computer Engineering and Computer Science University of Missouri-Columbia Columbia, MO 65211, U.S.A.

2. UMC – HCI seminar series 2 Query by Sketch -- an HCI topic? Human side: query behavior, retrieval result evaluation, training of semantics and database indexing. Computer side: query processing from human’s sketches, feature extraction from sketch objects, database search mechanisms, etc.

3. UMC – HCI seminar series 3 Some Applications and Systems Photo (natural scene images) IBM QBIC http://wwwqbic.almaden.ibm.com/http://wwwqbic.almaden.ibm.com/ GIS – Egenhofer at U. of Maine Spatial Query by sketch http://www.spatial.maine.edu/~max/RL.html http://www.spatial.maine.edu/~max/RL.html Medical Image – University of Missouri – Colmbia WebHIQS http://diglib1.cecs.missouri.edu:3243/FinalProj/index1.html (experimental website)

4. UMC – HCI seminar series 4 QBS in Diagnostic Image Databases Diagnoses often involve findings related to spatial relationship among lesions and landmarks. Pathologies could be detected based on many visual patterns that are “sketchable”, such as shapes of tumors, distribution of nodules, ect. Query: “Retrieve database images that have similar lesion- landmark location with the sketched query (image)”

5. UMC – HCI seminar series 5 Query methods in Diagnostic Image DBS Sketch on a blank sheet – users are provided with a GUI for drawing organs, landmarks, and lesions. Sketch on an organ template – users are provided with a template of an organ and a GUI for drawing landmarks and lesions. Sketch on an existing image – users are provided with a real diagnostic image and a GUI for drawing landmarks and lesions.

6. UMC – HCI seminar series 6 Research opportunity – Feature Extraction from Sketches Shapes from sketch objects, texture from sketched regions, gray scale from sketched vicinity, etc. Domain expertise will help to design computer vision algorithms for this purpose. Forming a feature vector for image retrieval. Searching a high-dimensional database for retrieving the most similar sketches?

7. UMC – HCI seminar series 7 Research opportunity – Spatial Modeling from Sketches Anatomical landmarks in HRCT lung: Lung regions (Automatic Extraction) Fissures (Human in the loop) Lobe partitions (HIL/AE) Lesions vs. landmarks: Interior Adjacent Across

8. UMC – HCI seminar series 8 Research opportunity – Spatial Modeling from Sketches - 2 Why these three lesions should be retrieved? Query lesion Sketched by the user

9. UMC – HCI seminar series 9 Spatial Modeling from Sketches - 3 Degree of Truth for (inner-Adjacent, To-the-right) (0.34, 0.00)(0.15, 0.00)(0.55, 0.31)(1.00, 0.82) (1.00, 0.96)(0.83, 0.84)(0.21, 0.99)(0.00, 1.00) What about multiple landmarks with multiple lesions?

10. UMC – HCI seminar series 10 Research opportunity – Semantic Query from Sketches Semantic or so-called ontology is very subjective and user-oriented. Not good for “general purposed” database retrieval. GREAT for Sketches! The hope is: Sketches might be able to bring more desirable retrieval results from user’s aspects.

11. UMC – HCI seminar series 11 Semantic Query from Sketches - 2

12. UMC – HCI seminar series 12 Semantic Query from Sketches - 3 Sketch bronchial structures on an organ template and fill gray scales to different part of the object, such as bronchial Walls and lumens.

13. UMC – HCI seminar series 13 Semantic Query from Sketches - 4 Users sketch on many training images and form a semantic tree structure for future retrieval.

14. UMC – HCI seminar series 14 Semantic Query from Sketches - 5 For fuzzy gangs, this looks familiar. We can form membership functions for the semantic terms by using user’s Sketches.

15. UMC – HCI seminar series 15 Semantic Query from Sketches - 6 Ranking database images by semantics

16. UMC – HCI seminar series 16 Semantic Query from Sketches - 6

17. UMC – HCI seminar series 17 Research opportunity – Query Languages for Sketches SELECT I.iid FROM LungImageDatabase I WHERE SIM( SKETCH(Qid,GUI), I.FE(iid)) < threshold ORDER BY SIM; A possible SQL extension for QBS:

18. UMC – HCI seminar series 18 Summary Query by sketch is not new, however, it has high potential for many applications, especially in the areas of diagnostic image database, GIS, criminal investigation, etc. In the CECS department, we have many existing research projects that are highly correlated to the concept of QBS. We should have some sort of synergistic efforts to have some interesting projects in the near future.