1. ProbExplorer: Uncertainty-guided Exploration and Editing of Probabilistic Medical Image Segmentation Ahmed Saad 1,2, Torsten Möller 1, and Ghassan Hamarneh 2 1 Graphics, Usability, and Visualization (GrUVi) Lab, 2 Medical Image Analysis Lab (MIAL), School of Computing Science, Simon Fraser University, Canada

2. Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 2

3. Ahmed Saad ProbExplorer Medical image segmentation Partitioning the image into disjoint regions of homogeneous properties Useful for statistical analysis, diagnosis, and treatment evaluation Medical Image Segmentation 3

4. Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data Magnetic Resonance Imaging Positron Emission Tomography 4

5. Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 5

6. Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 6 Patient 1 Patient 2Patient 3Patient 4

7. Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 4D CTdPET DTMRI 7

8. Ahmed Saad ProbExplorer Segmentation output CrispProbabilistic (Fuzzy) 70% 20% 10% Putamen White matter Grey matter Putamen 8 Max

9. Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 9

10. Ahmed Saad ProbExplorer Goal Given probabilistic segmentation results, we will allow expert users to visually examine regions of segmentation uncertainty to – Highlight suspicious regions (e.g. tumors) – Correct misclassification results without re- running the segmentation 10

11. Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 11

12. Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 12

13. Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 13

14. Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 14 BeforeAfter

15. Ahmed Saad ProbExplorer Preprocessing A probabilistic vector field Sort 15

16. Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 16

17. Ahmed Saad ProbExplorer Renal dynamic SPECT 4D image of size 64 x 64 x 32 with 48 time steps with an isotropic voxel size of (2 mm) 3 17 Raw data Crisp segmentation

18. Ahmed Saad ProbExplorer Uncertainty interaction overview widget 18   ?

19. Ahmed Saad ProbExplorer Selection of normal behavior 19

20. Ahmed Saad ProbExplorer Selection of abnormal behavior 20

21. Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 21

22. Ahmed Saad ProbExplorer Uncertainty-based segmentation editing 22 Ground truthOverestimationUnderestimation

23. Ahmed Saad ProbExplorer Synthetic example 23 No noise no PVE Ground truth Observed = noise + PVE Current segmentation

24. Ahmed Saad ProbExplorer Synthetic example: push action Push action Source set Destination set 24

25. Ahmed Saad ProbExplorer Synthetic example: push action 25 is the first best guess 0.4 0.3 0.2 Swap 0.3 0.4

26. Ahmed Saad ProbExplorer Dynamic PET brain 26

27. Ahmed Saad ProbExplorer Overestimated putamen 27 Ground truth Overestimated Putamen

28. Ahmed Saad ProbExplorer Uncertainty interaction overview widget 28

29. Ahmed Saad ProbExplorer Dynamic PET brain 29

30. Ahmed Saad ProbExplorer Dynamic PET brain Push action Putamen Background Skull Grey matter Cerebellum Source set Destination set 30

31. Ahmed Saad ProbExplorer Dynamic PET brain 31 After 2 editing actions

32. Ahmed Saad ProbExplorer More (Saad et al., EuroVis10) 32 Selection

33. Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 33

34. Ahmed Saad ProbExplorer Bayesian perspective LikelihoodPriorPosterior 34

35. Ahmed Saad ProbExplorer Framework Atlas construction Shape prior Likelihood Appearance prior Likelihood Images Expert binary segmentations Probabilistic shape prior Probabilistic appearance prior Population representative image New image New probabilistic segmentation Image-to-Image registration Aligned likelihood 35

36. Ahmed Saad ProbExplorer Mathematical notations 36

37. Ahmed Saad ProbExplorer Algorithm demonstration using synthetic example Piecewise constantBlurring Noise 100 noise realizations and random translations 37

38. Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 38

39. Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 39

40. Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 40

41. Ahmed Saad ProbExplorer Atlas construction: Appearance prior modeling 41

42. Ahmed Saad ProbExplorer Mixture of Gaussians Other probabilistic segmentation techniques can be used, e.g. Random walker, Probabilistic SVM, etc. Likelihood 42

43. Ahmed Saad ProbExplorer Abnormal cases 43

44. Ahmed Saad ProbExplorer Abnormal shape DataMaximum likelihood Selection 44

45. Ahmed Saad ProbExplorer Abnormal shape Data Selection Maximum likelihood 45

46. Ahmed Saad ProbExplorer Abnormal appearance Data Selection Maximum likelihood 46

47. Ahmed Saad ProbExplorer Abnormal shape and appearance Data Selection Maximum likelihood 47

48. Ahmed Saad ProbExplorer Misclassification correction Dice: 0.32Dice: 0.75 48

49. Ahmed Saad ProbExplorer More (Saad et al., IEEEVis10) 49

50. Ahmed Saad ProbExplorer User evaluation Our clinical collaborators showed how ProbExplorer can be used to achieve highly accurate segmentation from a very noisy dSPECT renal study (Humphries et al. IEEE Nuclear Science Symposium/Medical Image Conference 2009) 50

51. Ahmed Saad ProbExplorer Conclusion ProbExplorer: a framework for the analysis and visualization of probabilistic segmentation results We provided a number of visual data analysis widgets to reveal the different class interactions that are usually hidden by a simple crisp visualization 51

52. Ahmed Saad ProbExplorer Future work Spatial dependency between voxels during interactive editing Investigate the behavior of the resulting probabilistic results from different segmentation techniques Multi-structure atlas Registration uncertainty visualization 52

53. Ahmed Saad ProbExplorer Acknowledgements Natural Sciences and Engineering Research Council of Canada (NSERC) Prof. Vesna Sossi, Prof. Anna Celler, Thomas Humphries, and Prof. Manfred Trummer 53

54. 54 Ahmed Saad www.sfu.ca/~aasaad