1. I.R.SNApp Image Reconstruction and Segmentation for Neurosurgery Applications May09-10 Aaron Logan Dylan Reid William Lim Kyungchul Song Faculty Adviser and Client: Dr. Namrata Vaswani

2. © 2009, I.R.SNApp – all rights reserved. 2 Project Planning

3. © 2009, I.R.SNApp – all rights reserved. 3 Problem Introduction  MRI:  High quality  Few harmful effects on patient  Good candidate for image-assisted surgery  Requires real-time capability  Problem: current MRI is slow  Slow acquisition  Slow reconstruction  Problem: slow and inefficient segmentation (image processing) algorithms  Difficult to automatically analyze images for anomalies

4. © 2009, I.R.SNApp – all rights reserved. 4 Client Needs  MR Image Reconstruction using KFCS  Kalman Filtered Compressed Sensing  CS: relies on data sparseness (in some domain) to estimate signal using fewer observations/ measurements  Fewer measurements  faster reconstruction  Port existing MATLAB reconstruction prototype algorithm for runtime improvements  Develop algorithm for sequential image segmentation  Sequentially segment deforming objects (ROI's) from images  Utilize prior knowledge about shape change dynamics to segment noisy/low contrast imagery.  Use only current and past images in sequence

5. © 2009, I.R.SNApp – all rights reserved. 5 System Concept Diagram Image Reconstruction k-space (frequency domain) data – from MRI machine or simulated Image Segmentation Reconstructed MR image Region of interest contour data

6. © 2009, I.R.SNApp – all rights reserved. 6 Operating Environment & User Interface  Operating Environment  High-performance computing  Future work on this project may require parallel processing  User Interface  Command prompt  Simple GUI for segmentation  User input to choose ROI location to seed algorithm  Allow user to select various options  Visual image processing feedback

7. © 2009, I.R.SNApp – all rights reserved. 7 Project Requirements  Functional Requirements  FR-1: Algorithms shall output correct data. “Correctness” shall be determined by a certain error bound.  FR-2: Ported version of reconstruction algorithm shall run significantly faster than MATLAB prototype version.  Non-functional Requirements  NFR-1: The reconstruction program shall be written in C/C++.  NFR-2: The segmentation program shall be written in MATLAB.  NFR-3: The source code shall be well documented.

8. © 2009, I.R.SNApp – all rights reserved. 8 Deliverables  A complete software system that 1.takes raw MRI data, 2.reconstructs the data into images, and 3.performs segmentation on those images.  Attributes:  C/C++ code, well documented and working  Executable version of the code (MEX)  A basic GUI to assist segmentation process

9. © 2009, I.R.SNApp – all rights reserved. 9 Resource Requirements  C compilers  MinGW  LCC  C development environment – Eclipse & CDT  MATLAB, Image Processing Toolbox  GUI development environment – GUIDE  Test data  MR images  All can be obtained without cost

10. © 2009, I.R.SNApp – all rights reserved. 10 Project Risks  Segmentation  Very data-dependent  Image contrast, focus/resolution of objects, etc. influence final results  Mitigated with good image preprocessing – Contrast correction/normalization, filtering  “Good” segmentation algorithm difficult to define objectively  Reconstruction  KFCS algorithm could cause PCs to crash, resulting in data loss  Eliminated by testing only small images on PCs for framework testing; performing batch tests on full-size images on Linux clusters  Raw data acquisition  “Raw” MRI data only exists internal to the machine  HIPAA privacy laws may pose issue with acquisition

11. © 2009, I.R.SNApp – all rights reserved. 11 Challenges  Little background in:  C programming  Efficient algorithms  Debugging in C is quite tedious  No background in numerical computing

12. © 2009, I.R.SNApp – all rights reserved. 12 Work Breakdown  Image Reconstruction  Primarily C code development  Aaron Logan, William Lim  Image Segmentation  Investigation and development of segmentation algorithm  GUI development  MATLAB - based programming  Kyungchul Song, Dylan Reid  More team collaboration second semester

13. © 2009, I.R.SNApp – all rights reserved. 13 System Design Image Reconstruction

14. © 2009, I.R.SNApp – all rights reserved. 14 Design Approach  Profiled the MATLAB prototype to find most used functions  Results:

15. © 2009, I.R.SNApp – all rights reserved. 15 System Requirements  Specifications are determined with two goals in mind:  Runtime improvements  about 3 times faster than the MATLAB prototype  Data accuracy of the results compared with prototype: goal of 10 -5, i.e.

16. © 2009, I.R.SNApp – all rights reserved. 16 Functional Decomposition & System Analysis KFCS algorithm

17. © 2009, I.R.SNApp – all rights reserved. 17 Functional Decomposition & System Analysis  fnlCgv() – L1-penalized nonlinear conjugate gradient reconstruction  Developed by Michael Lustig (Stanford) in 2007 fnlcgv gXFMgTVgObj preobjective objective wGradient

18. © 2009, I.R.SNApp – all rights reserved. 18 Functional Decomposition & System Analysis  Components (see next slide):  Image reconstruction  kfcsfull - KFCS  fnlcgv – L1-penalized nonlinear conjugate gradient reconstruction  mexFunction() and convertparams()  Math functions – for example:  Indexing and subscripted assignment functions  Array dynamic allocation and deallocation functions  Warning and error messages

19. © 2009, I.R.SNApp – all rights reserved. 19 Functional Decomposition & System Analysis  Call graph (all functions)  Generated with Doxygen

20. © 2009, I.R.SNApp – all rights reserved. 20 I/O Specification  kfcsfull  Inputs: T_init, A, Q, y, FEN, Pi0, invR, params  Outputs: x_kfcsfull, x_kfcsfull2, T_kfcsfull, S_kfcsfull  fnlcgv  Inputs: x0, params  Output: x (actually – only necessary for MEX function)

21. © 2009, I.R.SNApp – all rights reserved. 21 Detailed Design Image Reconstruction

22. © 2009, I.R.SNApp – all rights reserved. 22 Software Details  Tradeoff: runtime efficiency vs. code efficiency // Compute matrix product mult_AB(DoubleArray *A, DoubleArray *B) { if (A and B are both real) { do something… } else if (A is real and B is complex) { do something else… } else if (A is complex and B is real) { do something else… } else // A and B are both complex { do something else… }

23. © 2009, I.R.SNApp – all rights reserved. 23 Software Details – Language and Tools  Language: C, not C++  Runs faster than C++: no overhead for OOP  Easier to learn; simple yet powerful  Development Environments  Eclipse and CDT - for source code development and management  MATLAB – for MEX file building  Compilers  MinGW gcc – used with Eclipse for heavy development  LCC (Local C Compiler) – used for compiling to build MEX files

24. © 2009, I.R.SNApp – all rights reserved. 24 Software Details – MEX Files  Advantages  Simplifies testing and comparison  Disadvantages/ limitations  Subject to MATLAB memory management practices  Unnecessary (costly) initialization of arrays  Compilation of functions in MATLAB  mex kfcsfull.c otherfile1.c otherfile2.c …  Builds executable library to be called by MATLAB  First file in list contains mexFunction()

25. © 2009, I.R.SNApp – all rights reserved. 25 Software Details – Data Representation  All matrices are stored in column-first form  All matrices & vectors, real or complex stored in DoubleArray objects with size data: typedef struct { double *re, *im; int M, N; } DoubleArray;  Data representation: double precision (64 bits)  eps in MATLAB = DBL_EPSILON in C = 2.2204E-16

26. © 2009, I.R.SNApp – all rights reserved. 26 Test Plan  Automated test scripts  Generate / read program input  Run C and MATLAB algorithms in succession on same input  Compare output of each algorithm by differencing  Compute statistics on differences  Scripts written in MATLAB; access C algorithms through MEX files (see next slide)  Unit testing in Eclipse  Math functions – compared 8 digits of precision  Indexing functions – verified functional behavior  Sorting and set math function – verified functional behavior

27. © 2009, I.R.SNApp – all rights reserved. 27 Test Specification MATLAB source code C source code Image set: x Reconstructed image set: k-space data: y MEX function (MATLAB/C interface) MATLAB reconstruction algorithms Generate k- space data (2-D FFT) Ported C reconstruction algorithms Compare output data; capture timing

28. © 2009, I.R.SNApp – all rights reserved. 28 Fnlcgv_ported Testing Results  Time comparison

29. © 2009, I.R.SNApp – all rights reserved. 29 Fnlcgv_ported Testing Results  Error % comparison for KFCS x data

30. © 2009, I.R.SNApp – all rights reserved. 30 Fnlcgv_ported Testing Results  Error % comparison for CS x data

31. © 2009, I.R.SNApp – all rights reserved. 31 System Design Image Segmentation

32. © 2009, I.R.SNApp – all rights reserved. 32 System Requirements  Develop segmentation algorithm in MATLAB  Achieve a contour that is correct  Includes only the ROI, no extraneous / artifact regions

33. © 2009, I.R.SNApp – all rights reserved. 33 System Requirements  Achieve a contour that is correct (cont.)  Correct by visual inspection  Good contour with fine width, fine precision  ~1 pixel width (8-connected) right wrong  Runtime upper bound: 1 sec/frame

34. © 2009, I.R.SNApp – all rights reserved. 34 System Overview  A system that takes sparse MR images as inputs  Brain  Other organs  Segments them by using an image seed from user input  Final goal of segmentation:  Simplify and change the representation of an image into something that is more meaningful and easier to analyze  Extract exact location of regions of interest within image

35. © 2009, I.R.SNApp – all rights reserved. 35 Functional Decomposition Convert to grayscale Extract ROI contour Contrast enhancement Threshold image (convert to binary) Fill holes Morphological Opening Select ROI Overlay contour on original image

36. © 2009, I.R.SNApp – all rights reserved. 36 Detailed Design Image Segmentation

37. © 2009, I.R.SNApp – all rights reserved. 37 Software Specification  Software Specification  MATLAB Version 7.6 (R2008a)  Image Processing Toolbox, Version 6.1

38. © 2009, I.R.SNApp – all rights reserved. 38 Input / Output Specification  Input:  Dynamic MR Image sequence  File format: Jpeg & Bitmap & DICOM 1  Size: no constraint  Grayscale  Directory information  Indices of images to process  Output:  MATLAB display:  current segmented image  montage of the original and segmented images  list of image files that have been segmented  File output:  DICOM of segmented image  CSV or MAT file of the contour pixel coordinates [1] (Digital Imaging and Communications in Medicine)

39. © 2009, I.R.SNApp – all rights reserved. 39 Test Specification  Visual inspection of output contour  Simple and most robust method of checking that output meets requirements  Speed (throughput time) of the system  Measured using the timing functions available with MATLAB  tic and toc stopwatch functions  clock  MATLAB Profiler

40. © 2009, I.R.SNApp – all rights reserved. 40 Segmentation System Structure GUI display update Seed for next image segmentation Filenames of images to segment Segmentation settings GUI Segmentation Driver Image Segmentation Algorithm User Input – first segmentation seed Segmented image (ROI highlighted) ROI contour data Current image Seed for current segmentation

41. © 2009, I.R.SNApp – all rights reserved. 41 Segmentation Algorithm (1/6)  Step 1  Threshold the image by using im2bw  Threshold found by graythresh

42. © 2009, I.R.SNApp – all rights reserved. 42 Segmentation Algorithm (2/6)  Step 2  Fill in the holes by using: imfill(BW, ‘holes’)

43. © 2009, I.R.SNApp – all rights reserved. 43 Segmentation Algorithm (3/6)  Step 3  Removes small objects by: bwareaopen

44. © 2009, I.R.SNApp – all rights reserved. 44 Segmentation Algorithm (4/6)  Step 4  Pick the region by using image seed data  bwselect

45. © 2009, I.R.SNApp – all rights reserved. 45 Segmentation Algorithm (5/6)  Step 5  Find the edge by using: bwperim

46. © 2009, I.R.SNApp – all rights reserved. 46 Segmentation Algorithm (6/6)  Step 6  Overlay the contour onto the original image

47. © 2009, I.R.SNApp – all rights reserved. 47 Conclusions  Helped our adviser with her research  Learned a lot about C programming, numerical computing and it challenges, and image processing  Future work:  Implementation of reconstruction system entirely in C  Parallelization of the reconstruction code and further optimization

48. © 2009, I.R.SNApp – all rights reserved. 48 Questions and Discussion