1. 1 ITK Lecture 8 - Neighborhoods Methods in Image Analysis CMU Robotics Institute 16-725 U. Pitt Bioengineering 2630 Spring Term, 2006 Methods in Image Analysis CMU Robotics Institute 16-725 U. Pitt Bioengineering 2630 Spring Term, 2006 Damion Shelton

2. 2 Goals for this lecture  Understand what a neighborhood is and and the different ways of accessing pixels using one  Use neighborhoods to implement a convolution/correlation filter  Understand what a neighborhood is and and the different ways of accessing pixels using one  Use neighborhoods to implement a convolution/correlation filter

3. 3 What is a neighborhood?  You may already be familiar with the concept of pixels having neighbors  Standard terminology in 2D image processing will refer to the 4 neighborhood (N,E,S,W) and the 8 neighborhood (4 neighborhood + NE, SE, SW, NW)  You may already be familiar with the concept of pixels having neighbors  Standard terminology in 2D image processing will refer to the 4 neighborhood (N,E,S,W) and the 8 neighborhood (4 neighborhood + NE, SE, SW, NW)

4. 4 Neighborhoods in ITK  ITK carries this concept a bit further  A neighborhood can be any collection of pixels that have a fixed relationship to the “center” based on offsets in data space  See 11.4 in the ITK Software Guide  ITK carries this concept a bit further  A neighborhood can be any collection of pixels that have a fixed relationship to the “center” based on offsets in data space  See 11.4 in the ITK Software Guide

5. 5 Neighborhoods in ITK, cont.  In general, the neighborhood is not completely arbitrary  Neighborhoods are rectangular, defined by a “radius” in N-dimensions  ShapedNeighborhoods are arbitrary, defined by a list of offsets from the center  The first form is most useful for mathematical morphology kinds of operations, convolution, etc.  In general, the neighborhood is not completely arbitrary  Neighborhoods are rectangular, defined by a “radius” in N-dimensions  ShapedNeighborhoods are arbitrary, defined by a list of offsets from the center  The first form is most useful for mathematical morphology kinds of operations, convolution, etc.

6. 6 Neighborhood iterators  The cool & useful thing about neighborhoods is that they can be used with neighborhood iterators to allow efficient access to pixels “around” a target pixel in an image

7. 7 Neighborhood iterators  Remember that I said access via pixel indices was slow?  Get current index = I  Upper left pixel index I UL = I - (1,1)  Get pixel at index I UL  Neighborhood iterators solve this problem by doing pointer arithmetic based on offsets  Remember that I said access via pixel indices was slow?  Get current index = I  Upper left pixel index I UL = I - (1,1)  Get pixel at index I UL  Neighborhood iterators solve this problem by doing pointer arithmetic based on offsets

8. 8 Neighborhood layout  Neighborhoods have one primary parameter, their “radius” in N- dimensions  The side length along a particular dimension i is 2*radius i + 1  Note that the side length is always odd because the center pixel always exists  Neighborhoods have one primary parameter, their “radius” in N- dimensions  The side length along a particular dimension i is 2*radius i + 1  Note that the side length is always odd because the center pixel always exists

9. 9 A 2x1 neighborhood in 2D 01234 56789 1011121314

10. 10 Stride  Neighborhoods have another parameter called stride which is the spacing (in data space) along a particular axis between adjacent pixels in the neighborhood  In the previous numbering scheme, stride in Y is amount then index value changes when you move in Y  In our example, Stride x = 1, Stride y = 5  Neighborhoods have another parameter called stride which is the spacing (in data space) along a particular axis between adjacent pixels in the neighborhood  In the previous numbering scheme, stride in Y is amount then index value changes when you move in Y  In our example, Stride x = 1, Stride y = 5

11. 11 Neighborhood pixel access  The numbering on the previous page is important! It’s how you access that particular pixel when using a neighborhood iterator  This will be clarified in a few slides...  The numbering on the previous page is important! It’s how you access that particular pixel when using a neighborhood iterator  This will be clarified in a few slides...

12. 12 NeighborhoodIterator access  Neighborhood iterators are created using:  The radius of the neighborhood  The image that will be traversed  The region of the image to be traversed  Their syntax largely follows that of other iterators (++, IsAtEnd(), etc.)  Neighborhood iterators are created using:  The radius of the neighborhood  The image that will be traversed  The region of the image to be traversed  Their syntax largely follows that of other iterators (++, IsAtEnd(), etc.)

13. 13 Neighborhood pixel access, cont. 1.21.31.81.41.1 1.81.10.71.0 2.11.91.71.42.0 Let’s say there’s some region of an image that has the following pixel values

14. 14 Pixel access, cont.  Now assume that we place the neighborhood iterator over this region and start accessing pixels  What happens?  Now assume that we place the neighborhood iterator over this region and start accessing pixels  What happens?

15. 15 Pixel access, cont. 1.2 0 1.3 1 1.8 2 1.4 3 1.1 4 1.8 5 1.1 6 0.7 7 1.0 8 1.0 9 2.1 10 1.9 11 1.7 12 1.4 13 2.0 14 myNeigh.GetPixel(7) returns 0.7 so does myNeigh.GetCenterPixel()

16. 16 Pixel access, cont. Next, let’s get the length of the iterator and the stride length Size() returns the #pixels in the neighborhood unsigned int c = iterator. Size () / 2; GetStride returns the stride of dimension N unsigned int s = iterator. GetStride(1); Next, let’s get the length of the iterator and the stride length Size() returns the #pixels in the neighborhood unsigned int c = iterator. Size () / 2; GetStride returns the stride of dimension N unsigned int s = iterator. GetStride(1);

17. 17 Pixel access, cont. 1.2 0 1.3 1 1.8 2 1.4 3 1.1 4 1.8 5 1.1 6 0.7 7 1.0 8 1.0 9 2.1 10 1.9 11 1.7 12 1.4 13 2.0 14 myNeigh.GetPixel(c) returns 0.7 myNeigh.GetPixel(c-1) returns 1.1

18. 18 Pixel access, cont. 1.2 0 1.3 1 1.8 2 1.4 3 1.1 4 1.8 5 1.1 6 0.7 7 1.0 8 1.0 9 2.1 10 1.9 11 1.7 12 1.4 13 2.0 14 myNeigh.GetPixel(c-s) returns 1.8 myNeigh.GetPixel(c-s-1) returns 1.3

19. 19 The ++ method  In ImageRegionIterators, the ++ method moves the focus of the iterator on a per pixel basis  In NeighborhoodIterators, the ++ method moves the center pixel of the neighborhood and therefore implicitly shifts the entire neighborhood  In ImageRegionIterators, the ++ method moves the focus of the iterator on a per pixel basis  In NeighborhoodIterators, the ++ method moves the center pixel of the neighborhood and therefore implicitly shifts the entire neighborhood

20. 20 Does this sound familiar?  If I say:  I have a region of interest defined by a certain radius around a center pixel  The ROI is symmetric  I move it around an image  What does this sound like?  If I say:  I have a region of interest defined by a certain radius around a center pixel  The ROI is symmetric  I move it around an image  What does this sound like?

21. 21 Convolution (ahem, correlation)! To do convolution we need 3 things: 1. A kernel 2. A way to access a region of an image the same size as the kernel 3. A way to compute the inner product between the kernel and the image region To do convolution we need 3 things: 1. A kernel 2. A way to access a region of an image the same size as the kernel 3. A way to compute the inner product between the kernel and the image region

22. 22 Item 1 - The kernel  A NeighborhoodOperator is a set of pixel values that can be applied to a Neighborhood to perform a user-defined operation (i.e. convolution kernel, morphological structuring element)  NeighborhoodOperator is derived from Neighborhood  A NeighborhoodOperator is a set of pixel values that can be applied to a Neighborhood to perform a user-defined operation (i.e. convolution kernel, morphological structuring element)  NeighborhoodOperator is derived from Neighborhood

23. 23 Item 2 - Image access method  We already showed that this is possible using the neighborhood iterator  Just be careful setting it up so that it’s the same size as your kernel  We already showed that this is possible using the neighborhood iterator  Just be careful setting it up so that it’s the same size as your kernel

24. 24 Item 3 - Inner product method  The NeighborhoodInnerProduct computes the inner product between two neighborhoods  Since NeighborhoodOperator is derived from Neighborhood, we can compute the IP of the kernel and the image region  The NeighborhoodInnerProduct computes the inner product between two neighborhoods  Since NeighborhoodOperator is derived from Neighborhood, we can compute the IP of the kernel and the image region

25. 25 Good to go? 1. Create an interesting operator to form a kernel 2. Move a neighborhood through an image 3. Compute the IP of the operator and the neighborhood at each pixel in the image Voila - convolution in N-dimensions 1. Create an interesting operator to form a kernel 2. Move a neighborhood through an image 3. Compute the IP of the operator and the neighborhood at each pixel in the image Voila - convolution in N-dimensions

26. 26 Inner product example itk::NeighborhoodInnerProduct IP; itk::DerivativeOperator operator ; operator->SetOrder(1); operator->SetDirection(0); operator->CreateDirectional(); itk::NeighborhoodIterator iterator(operator->GetRadius(), myImage, myImage->GetRequestedRegion()); itk::NeighborhoodInnerProduct IP; itk::DerivativeOperator operator ; operator->SetOrder(1); operator->SetDirection(0); operator->CreateDirectional(); itk::NeighborhoodIterator iterator(operator->GetRadius(), myImage, myImage->GetRequestedRegion());

27. 27 Inner product example, cont. iterator.SetToBegin(); while ( ! iterator. IsAtEnd () ) { std::cout << "Derivative at index " << iterator.GetIndex () << is << IP(iterator, operator) << std::endl; ++iterator; } iterator.SetToBegin(); while ( ! iterator. IsAtEnd () ) { std::cout << "Derivative at index " << iterator.GetIndex () << is << IP(iterator, operator) << std::endl; ++iterator; }

28. 28 Note  No explicit reference to dimensionality in neighborhood iterator   easy to make N-d  No explicit reference to dimensionality in neighborhood iterator   easy to make N-d

29. 29 This suggests a filter...  NeighborhoodOperatorImageFilter wraps this procedure into a filter that operates on an input image  So, if the main challenge is coming up with an interesting neighborhood operator, ITK can do the rest  NeighborhoodOperatorImageFilter wraps this procedure into a filter that operates on an input image  So, if the main challenge is coming up with an interesting neighborhood operator, ITK can do the rest

30. 30 Your arch-nemesis... image boundaries  One obvious problem with inner product techniques is what to do when you reach the edge of your image  Is the operation undefined?  Does the image wrap?  Should we assume the rest of the world is empty/full/something else?  One obvious problem with inner product techniques is what to do when you reach the edge of your image  Is the operation undefined?  Does the image wrap?  Should we assume the rest of the world is empty/full/something else?

31. 31 ImageBoundaryCondition  Subclasses of itk::ImageBoundaryCondition can be used to tell neighborhood iterators what to do if part of the neighborhood is not in the image

32. 32 ConstantBoundaryCondition  The rest of the world is filled with some constant value of your choice  The default is 0  Be careful with the value you choose - you can (for example) detect edges that aren’t really there  The rest of the world is filled with some constant value of your choice  The default is 0  Be careful with the value you choose - you can (for example) detect edges that aren’t really there

33. 33 PeriodicBoundaryCondition  The image wraps, so that if I exceed the length of a particular axis, I wrap back to 0 and start over again  If you enjoy headaches, imagine this in 3D  This isn’t a bad idea, but most medical images are not actually periodic  The image wraps, so that if I exceed the length of a particular axis, I wrap back to 0 and start over again  If you enjoy headaches, imagine this in 3D  This isn’t a bad idea, but most medical images are not actually periodic

34. 34 ZeroFluxNeumannBoundary Condition  I am not familiar with how this functions  The documentation states that it’s useful for solving certain classes of differential equations  A quick look online suggests a thermodynamic motivation  I am not familiar with how this functions  The documentation states that it’s useful for solving certain classes of differential equations  A quick look online suggests a thermodynamic motivation

35. 35 Using boundary conditions  With NeighborhoodOperatorImageFilter, you can call OverrideBoundaryCondition

36. 36 SmartNeighborhoodIterator  This is the iterator that’s being used internally by the previous filter; you can specify its boundary behavior using OverrideBoundaryCondition too  In general, I would suggest using the “smart” version - bounds checking is good!  This is the iterator that’s being used internally by the previous filter; you can specify its boundary behavior using OverrideBoundaryCondition too  In general, I would suggest using the “smart” version - bounds checking is good!

37. 37 An aside: numeric traits  This has nothing to do with Neighborhoods but is good to know  Question: given some arbitrary pixel type, what do we know about it from a numerics perspective?  This has nothing to do with Neighborhoods but is good to know  Question: given some arbitrary pixel type, what do we know about it from a numerics perspective?

38. 38 itk::NumericTraits  NumericTraits is class that’s specialized to provide information about pixel types  Examples include:  min and max values  IsPositive(), IsNegative()  Definitions of Zero and One  NumericTraits is class that’s specialized to provide information about pixel types  Examples include:  min and max values  IsPositive(), IsNegative()  Definitions of Zero and One

39. 39 Using traits  What’s the maximum value that can be represented by an unsigned char?  itk::NumericTraits ::max()  Look at vnl_numeric_limits for more data that can be provided  What’s the maximum value that can be represented by an unsigned char?  itk::NumericTraits ::max()  Look at vnl_numeric_limits for more data that can be provided