1. ITK Video ITK Extensions for Video Processing
Amitha Perera, Patrick Reynolds, Matt Leotta,
Luis Ibanez, Gabe Hart
Real-Time Image Capture for ITK through a Video Grabber
Kevin Cleary
Methods in Medical Image Analysis: An ITK-Based Course with Deliverable Algorithms that extends and evaluates ITK while broadening its developer base
John Galeotti

2. Goal Make video a first-class citizen of ITK Video I/O
Allow ITK to be used with Computer Vision libraries (OpenCV, vxl, etc.)
Allow other libraries to be used with ITK
Implement ITK filters using other libraries
Ensure framework allows real-world video problems to be researched
E.g. don't try solve hard real-time, but make sure real-time issues can be solved

3. Assumptions and Choices
Time is not another dimension
Video != 3-d pixel volume
Unbounded (live cameras)
Memory (2 hrs 1920x1080 color video = 44 GB)
On-line processing (streaming applications)
We haven't invented time travel yet
No explicit tagging of color spaces and other metadata (except for time)
Video readers will produce grayscale or RGB
Similar to CT and Hounsfield

4. Status: Real-time capture
Using avacap for frame grabbing (GPL)
Created an ITK version which is not GPL
Ring buffer implemented (duplicated)

5. Status: Video Extensions
4-hr tutorial at CVPR
Presented: ITK; using ITK with OpenCV; using OpenCV with ITK; direct video support in ITK
~33 people stayed for whole session
Raffle ticket histogram:
1: 22
2: 6
3: 15
4: 5
5: 7
6: 4
7: 2

6. Video in ITK reader->SetFileName( argv[1] ); // video reader
writer->SetFileName( argv[2] ); // video writer
frameDifferenceFilter->SetFrameOffset(1);
videoCaster->SetImageFilter( imageCaster );
imageThresh->ThresholdBelow( 128 );
videoThresh->SetImageFilter( imageThresh );
imageCurvatureFlowFilter->SetTimeStep( 0.5 );
imageCurvatureFlowFilter->SetNumberOfIterations( 20 );
videoCurvatureFlowFilter->SetImageFilter( imageFilter );
videoCurvatureFlowFilter->SetInput( reader- >GetOutput() );
videoCaster->SetInput( videoFilter->GetOutput() );
frameDifferenceFilter->SetInput( videoCaster- >GetOutput() );
videoThresh->SetInput( frameDifferenceFilter- >GetOutput() );
writer->SetInput( videoThresh->GetOutput() );
writer->Update(); // generate output video frame by frame, without reading whole source video at once

7. New Classes

8. New classes Real-time capture:
A new video I/O classes wrapping a capture library
Course:
ITK video filters
ITK representations of non-pixel data (tracks, etc.)

9. Class Hierarchy itk::DataObject itk::ProcessObject itk::TemporalRegion
itk::RingBuffer
itk::TemporalProcessObject
itk::TemporalDataObject
itk::VideoFileWriter
itk::VideoIOBase
itk::ImageIOBase
itk::OpenCVVideoIO
itk::VXLVideoIO
itk::FileListVideoIO
itk::VideoStream
itk::VideoSource
itk::VideoFileReader
itk::VideoToVideoFilter
itk::FrameDifferenceVideoFilter
itk::FrameAverageVideoFilter
itk::DecimateVideoFilter
itk::ImageFilterToVideoFilterWrapper

10. Temporal Data Objects

11. itk::TemporalDataObject
itk::TemporalRegion
itk::TemporalDataObject
FrameStart
FrameDuration
LargestPossibleTemporalRegion
RequestedTemporalRegion
RealStart
BufferedTemporalRegion
RealDuration
DataObjectBuffer
itk::RingBuffer 1 3 2 5 4
...
itk::DataObject

12. std::map< frame num, … >
itk::VideoStream
itk::TemporalRegion
itk::VideoStream
FrameStart
FrameDuration
LargestPossibleTemporalRegion
RequestedTemporalRegion
RealStart
BufferedTemporalRegion
RealDuration
DataObjectBuffer
itk::Region
itk::RingBuffer
LargestPossibleSpatialRegionCache
LargestPossibleSpatialRegionCache
itk::Point
LargestPossibleSpatialRegionCache 1
std::map< frame num, … >
itk::Vector 3 2
OriginCache
itk::Matrix
SpacingCache 5 4
...
DirectionCache
itk::Image

13. Temporal Process Objects

14. itk::TemporalProcessObject
itk::TemporalDataObject
itk::TemporalDataObject
itk::TemporalDataObject
itk::TemporalDataObject
Input
Output n -1 n n +1 n n +1
UnitInputNumberOfFrames
UnitOutputNumberOfFrames
InputStencilCurrentFrameIndex
FrameSkipPerOutput

15. itk::VideoToVideoFilter
itk::TemporalDataObject
itk::TemporalDataObject
itk::VideoStream
itk::TemporalDataObject
Input
Output n -1 n n +1 n n +1
UnitInputNumberOfFrames
UnitOutputNumberOfFrames
InputStencilCurrentFrameIndex
FrameSkipPerOutput

16. itk::ImageToImageFilter
itk::VideoToVideoFilter
itk::ImageToImageFilter →GenerateData Process
GenerateData()
ThreadedGenerateData( … )

17. itk::VideoToVideoFilter
itk::VideoToVideoFilter →GenerateData Process
GenerateData()
TemporalStreamingGenerateData()
ThreadedGenerateData( … )

18. Pipeline Architecture

19. ITK Pipeline

20. UpdateOutputInformation
ITK Pipeline
UpdateOutputInformation
Reader
Filter A
Filter B
Update()

21. UpdateOutputInformation
ITK Pipeline
UpdateOutputInformation
Reader
Filter A
Filter B
Update()

22. UpdateOutputInformation
ITK Pipeline
UpdateOutputInformation
Reader
Filter A
Filter B
Update()

23. ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion
Filter A
Filter B
Update()

24. ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion
Filter A
Filter B
Update()

25. ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion
UpdateOutputData
Filter A
Filter B
Update()

26. ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion
UpdateOutputData
Filter A
Filter B
Update()

27. ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion
UpdateOutputData
Filter A
Filter B
Update()

28. ITK Video Pipeline

29. UpdateOutputInformation
ITK Video Pipeline
UpdateOutputInformation
0:1
Reader
3:1
Filter A
2:1
Filter B
Update()

30. UpdateOutputInformation
ITK Video Pipeline
UpdateOutputInformation
0:1
Reader
3:1
Filter A
2:1
Filter B
Update()

31. UpdateOutputInformation
ITK Video Pipeline
UpdateOutputInformation
0:1
Reader
3:1
Filter A
2:1
Filter B
Update()

32. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
3:1
Filter A
2:1
Filter B
Update()

33. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
3:1
Filter A
2:1
Filter B
Update()

34. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

35. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

36. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

37. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

38. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

39. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

40. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

41. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

42. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

43. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

44. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

45. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

46. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

47. ITK Video Pipeline UpdateOutputInformation 0:1 Reader
PropagateRequestedRegion
UpdateOutputData
3:1
Filter A
2:1
Filter B
Update()

48. OpenCV / VXL Bridges

49. Single Image Bridges itk::OpenCVImageBridge IplImage ↔ itk::Image
cv::Mat ↔ itk::Image
itk::VXLImageBridge (forthcoming)
vil_image ↔ itk::Image
vidl_frame ↔ itk::Image

50. Video Bridges itk::OpenCVVideoCapture
Subclass of OpenCV's cv::VideoCapture
Takes itk::VideoStream as input
Triggers pipeline attached to input if necessary
itk::vidl_itk_istream
Subclass of VXL's vidl_istream

51. IO Mechanism

52. IO Mechanism Itk::VideoFileReader Itk::VXLVideoIO
Itk::VXLVideoIOFactory
Itk::VideoStream
Itk::VideoIOFactory
Itk::VideoIOBase
Itk::ImageIOBase
Itk::OpenCVVideoIOFactory
Itk::OpenCVVideoIO
Itk::VideoFileWriter

53. Module Structure

54. Module Structure - Current
VideoCore
VideoFilters
VideoIO
Common
OpenCV
VXL
Common
OpenCV
VXL
Core
OpenCV
VXL
FileList
Directories
Modules / Groups
Video-Core-Common
Video-Filters-Common
Video-IO-Core
Video-Core-OpenCV
Video-Filters-OpenCV
Video-IO-OpenCV
Video-Core-VXL
Video-Filters-VXL
Video-IO-VXL
Video-IO-FileList
Group_VideoCommon
Group_VideoOpenCV
Group_VideoVXL

55. Module Structure – Desired (?)
VideoCore
VideoFilters
VideoIO
Common
OpenCV
VXL
Common
OpenCV
VXL
Core
OpenCV
VXL
FileList ? ? ?
Directories
Modules / Groups
Video-Core-Common
Video-Filters-Common
Video-IO-Core
Video-Core-OpenCV
Video-Filters-OpenCV
Video-IO-OpenCV
Video-Core-VXL
Video-Filters-VXL
Video-IO-VXL
Video-IO-FileList
Group_VideoCommon
Group_VideoOpenCV
Group_VideoVXL

56. Issues/ToDo RealTimeStamp is not a real time stamp I/O readers
Image sequence as a video (done via file list)
Multi-frame DICOM file as a video
Video capture
sync with Children's National on video reader and on using framework
Direct video I/O (using ffmpeg)
VideoStream → 3-d volume converter
Handle infinite duration
Implement timestamp-based flow (vs frame-based flow)
Implement video filter base classes for multiple inputs and outputs
Synchronization of multiple streams
Avoid memcpy where possible (when wrapping other libraries)
Ensure frame size is not an intrinsic problem
Working with SD video now, but will need to work with HD video
Implement filters in new video framework (e.g. using OpenCV)
Translators for other data structures (e.g. OpenCV sequences, tracks, etc.)

57. END