1. 12 August 2015 Chair for Computer Aided Medical Procedures & Augmented Reality Department of Computer Science | Technische Universität München Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Automatic Feature Generation for Endoscopic Image Classification Ulrich Klank

2. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 2 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Endoscopic Image Classification  Synchronization of surgeries by phases is essential for Workflow Project 1  Analyzing endoscopic images supports the classification of the current state into a phase. 1 A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, N. Navab -Recovery of surgical workflow without explicit models

3. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 3 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Task  How can these images be classified into surgical phases?  Or if they can’t be classified well enough, which features can be added to the rest of the information?

4. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 4 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Definition of a Feature  Feature as representation of discriminative information  Numerical vector calculated out of the image  Reduction of the dimension  Features  Basic: Gradient/Color/Motion Histogram  Integral image 1  Phase concruency 2 Gradient Histogram 1 Viola, Jones- Robust Real-time Object Detection 2 Kovesi - Image Features from Phase Congruency

5. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 5 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Feature Space  With a limit to algorithms with an output smaller or equal than the input we get a number of injective function about: depending on the dimension of the input vector d and the number of different values c which are possible in the input.  Exploration of this space with Genetic Programming 1  A method to explore a high dimensional space  Fitness function influences the resulting features 1 Feature Extraction from Multiple Data Sources Using Genetic Programming - John J. Szymanski,* Steven P. Brumby, Paul Pope, Damian Eads, Diana Esch-Mosher,Mark Galassi, Neal R. Harvey, Hersey D.W. McCulloch, Simon J. Perkins, Reid Porter,James Theiler, A. Cody Young, Jeffrey J. Bloch and Nancy David ( c d ) c d ( c d ¡ 1 ) 2

6. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 6 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Feature Generation by Genetic Programming 12 Mutation Evaluation Code 1 Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, Frank D. Francone - Genetic Programming: An Introduction 2 Koza - Genetic Programming

7. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 7 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Implementation Steps  Definition of a programming language (more)(more)  Designed for recombination and mutation  Designing a Framework to run programs  Translation of basic features in the new language  Evaluation of the fitness of programs  Recombination and Mutation

8. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 8 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu An Example as Proof of Concept  Implementation of a color histogram in the new language  Test on an easy sample, 6 images(10*10*RGB, jpeg) in 3 classes:  Dimension of input here (cd): 76.800

9. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 9 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example  The Fitness Function is the core of the system

10. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 10 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Comparison of several Programs P i P1P1 P1P1 P2P2 P2P2 …

11. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 11 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example A random selection of the labeled input I k several from of each class P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3

12. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 12 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Execution of the program with every selected input I k As result there are output vectors: P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23

13. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 13 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Compare each pair of output vectors of one Program, by the distance P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 o 11 ; o 12 ¢¢¢ o 23

14. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 14 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Select the maximum of these distances, only considering pairs of different classes: P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 Now we have two classes k, l which our feature separates argmax kl ( d i kl j k : c l ass 6 = l : c l ass )

15. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 15 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the example Subtract the distances in images of the classes k or l P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 f i = d i kl ¡ d i kk + d i ll 2 argmax kl ( d i kl j k : c l ass 6 = l : c l ass )

16. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 16 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu 3.1 Results in this Example New Program

17. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 17 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Fitness in the Example:  Input Program was calculating a Color Histogram (Output length 512 in ~2000 steps), resulting fitness: ~ 1.2 %  The new generated program calculated one number (in 30 steps) and got a fitness of: ~ 98.8 % P1P1 P1P1 P2P2 P2P2

18. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 18 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evolution Technique used in the Example  Selection of the two best Program and generation several children by these two methods:  One parental program: Mutation  Adding of random commands  Removing a single commands1  Duplicating an existing command  Switching commands  Two parental programs: Breeding  Mixing both, preferring blocks of a specific size  Adding random commands in the gaps between P1P1 P1P1 P 11 P1P1 P1P1 P2P2 P2P2 P12P12 P12P12

19. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 19 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Conclusion  The concept seems to work  Adaptation to workflow project’s data can start

20. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 20 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Steps to adapt it to the Workflow project  Adapt fitness function (more)(more)  Change Genetic rules (more)(more)  Optimize the framework  Generate C code out of the intermediate language

21. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 21 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Any Questions?

22. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 22 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Appendix

23. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 23 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Planned Improvements in the Genetic rules  Probability to survive should depend on the fitness of all other existing programs  Even a program with a low fitness should have a small change to survive  Also the reproduction should not be restricted to the best program Back

24. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 24 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Fitness  Actual fitness expresses:  Linear separation of two classes  Further improvements:  Static Analysis of Code 1  Semantic Checks  to prevent running in infinite loops  to prevent the execution of programs not using the input  …  Other Classifiers to evaluate the output 1 Colin G. Johnson-Deriving genetic programming fitness properties by static analysis Back

25. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 25 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Specification of the language  Assembler style with some high level functions  1 Opcode and 3 Arguments for one command  High level examples:  Reading of the input and its dimensions  For loops  Adding of more functions possible Back

26. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 26 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Scheme for the programming language OPCode1.Parameter2. Parameter3. Parameter 010 For counter to stack 0 From Constant 100 To Constant 65 End of the loop Constant 123 Read pixel Value to stack x From stack 20 y Constant … …

27. CAMP | Department of Computer Science | Technische Universität München | 12 August 2015 27 Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Generated Program (without nops) INPUTDIMENSION 0 0 4595026 SAVE 0 -1 0 INPUTDIMENSION 1 0 0 SAVE 23367 -1 0 FOR 0 -1 37 LOAD 4 0 0 LOAD 1 0 0 SAVE 5 -1 0 LOAD 5 0 0 POP 0 0 -1 READINPUT -1 -1 -1 LOAD 5 0 -1 POP 0 0 -1 PUSH 64 0 -1 SAVE 3 -1 0 PUSH 8 0 0 MULTIPLY 0 0 0 LOAD 3 0 0 PLUS 0 12604 0 PLUS 0 6457 0 PLUS 31190 0 0 JUMP 55 0 0 DIVIDE 0 0 0 PUSH 32 0 0 LOAD 1 -1 0 LOAD 2 0 0 RETURN 0 0 0 JUMP 0 0 0 FOR 8 520 58