1. Chair for Computer Aided Medical Procedures & Augmented Reality Department of Computer Science | Technische Universität München Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Automatic Feature Generation for Endoscopic Image Classification Ulrich Klank Nicolas Padoy Prof. Nassir Navab Supervisor: Advisor: 18 January 2007

2. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 2 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Overview  Endoscopic images  Differences  Similarities  Image feature generation using Genetic Programming  A low level approach  A high level approach  An example

3. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 3 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Endoscopic Images of two Surgical Phases: OP3 Images from the cutting and clipping phase (OP3) Images from the bag retraction phase (OP3)

4. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 4 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu …: OP1 Images from the bag retraction phase (OP1)Images from the cutting and clipping phase (OP1)

5. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 5 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Low-level Approach - Short Reminder Genetic Programming: combination of low-level operators PIXEL (With Parameters) PUSH… FOR (With Parameters) MUL LOAD (With Parameters) ADD Mutation Evaluation Code

6. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 6 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Low-level Approach - Results  Distributed evaluation of programs on several computers (up to 7)  Nearly 10.000 programs evaluated (~300 generations)  First results:  Characteristics of the best programs:  returning a short vector in a short time  Classification rate with a linear classifier is 62%  (64 images of 2 phases of 4 videos)

7. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 7 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Problems with the low-level Approach  ~40% of the programs contain major errors like  Infinite running time, stack overflow  No reference to the input image  Resulting programs still has structural similarity to the initial program. More generations needed  Evaluation of a programs is very slow due to the simulation of basic instructions on images How to improve this method?

8. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 8 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Existing Software: GENIE  GENIE Software published by:  Los Alamos National Laboratories  First publication ’97,  Commercial version in development  Genetic Programming for segmentation of images  Application example:  Segmentation of Medical Images Using a Genetic Algorithm by Payel Ghosh, Melanie Mitchell (’06)

9. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 9 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu The Step to a higher level approach Erode (With Parameters) Histogram (With Parameters) MinLocOriginal CannyEdge (With Parameters) Gradient x (With Parameters) Dilate (With Parameters) PIXEL (With Parameters) PUSH… FOR (With Parameters) MUL LOAD (With Parameters) ADD Replace the basic commands in a program by higher level operators: Examples for low-level operator : Examples for high-level operator :

10. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 10 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of a Program  Semantic checks  Input referred?  No infinite loops?  Execution with several inputs  16 images per phase  2 phases per video at the moment  4 videos used for evaluation  A fitness function with 2 components:  A classification of the phases by the output vectors  The average execution time per input

11. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 11 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Genetic Concept – Cross Over Erode (With Parameters) Histogram (With Parameters) MinLocOriginal CannyEdge (With Parameters) Gradient x (With Parameters) Dilate (With Parameters) Gauss (With Parameters) OriginalMaxOriginal PushImage (With Parameters) Gradient y (With Parameters) Histogram (With Parameters) Program 1 Program 2 New Program

12. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 12 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu High-level Approach Benefits  Faster evaluation  Reduced number of commands  Optimized basic image operations (OpenCV)  Resulting programs easier to understand

13. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 13 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Current Results:  Running time: 480 ms (in simulation)  Output length: a vector of 48 signed integer  Classification rate: 67%  Rate based on 512 testing images out of 4 videos and 2 phases  Number of generations needed: ~80

14. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 14 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Future Work  Completion of the available image operators  Extension to multi-phases classification  Comparison of the fitness function with a standard classifier  Comparison with several standard features  Features evaluation within the workflow segmentation system

15. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 15 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Thank you for your attention!