ITK Video ITK Extensions for Video Processing

Slides:

Advertisements

Similar presentations

Support for Time in ITK Patrick Reynolds Patrick Cheng John Galeotti Arnaud Gelas.

Advertisements

ITK Architecture Kitware Inc.. ITK Basics C++ Generic Programming Data Pipeline Multi-threading Streaming Exceptions Events / Observers Tcl, Python and.

National Alliance for Medical Image Computing Slicer3 plugins: Going Further Common architecture for interactive and batch processing.

National Alliance for Medical Image Computing Slicer3 plugins Common architecture for interactive and batch processing.

Embedded Streaming Media with GStreamer and BeagleBoard ESC-228 Presented by Santiago Nunez santiago.nunez (at) ridgerun.com.

SE263 Video Analytics Course Project Initial Report Presented by M. Aravind Krishnan, SERC, IISc X. Mei and H. Ling, ICCV’09.

Yingcai Xiao Chapter 4 The Visualization Pipeline.

Distributed Multimedia Systems

areaDetector Developments

Methods in Image Analysis 2004, Damion Shelton 1 ITK Lecture 6 - The Pipeline Methods in Image Analysis CMU Robotics Institute U. Pitt Bioengineering.

Foundations & Core in Computer Vision: A System Perspective Ce Liu Microsoft Research New England.

This work by John Galeotti and Damion Shelton, © , was made possible in part by NIH NLM contract# HHSN P, and is licensed under a Creative.

NA-MIC National Alliance for Medical Image Computing 3D Slicer Interactive Modules Steve Pieper, PhD.

Modeling Pixel Process with Scale Invariant Local Patterns for Background Subtraction in Complex Scenes (CVPR’10) Shengcai Liao, Guoying Zhao, Vili Kellokumpu,

HCI Final Project Robust Real Time Face Detection Paul Viola, Michael Jones, Robust Real-Time Face Detetion, International Journal of Computer Vision,

ADVISE: Advanced Digital Video Information Segmentation Engine

Real-Time Face Detection and Tracking Using Multiple Cameras RIT Computer Engineering Senior Design Project John RuppertJustin HnatowJared Holsopple This.

ICBV Course Final Project Arik Krol Aviad Pinkovezky.

AreaDetector and GStreamer Similarities and Differences Tom Cobb.

Group 4: Web based applications/ crowdsourcing Marcel Prastawa Ziv Yaniv Patrick Reynolds Stephen Aylward Sean Megason.

Kalman Tracking for Image Processing Applications Student : Julius Oyeleke Supervisor : Dr Martin Glavin Co-Supervisor : Dr Fearghal Morgan.

Harris Corner Detector on FPGA Rohit Banerjee Jared Choi : Parallel Computer Architecture and Programming.

NA-MIC National Alliance for Medical Image Computing Core 1b – Engineering Computational Platform Jim Miller GE Research.

T EACHING ITK: T HOUGHTS FOR V ERSION 4 John Galeotti February 3, 2011.

How to Choose Frame Grabber …that’s right for your application Coreco Imaging.

The Pipeline Processing Framework LSST Applications Meeting IPAC Feb. 19, 2008 Raymond Plante National Center for Supercomputing Applications.

Reconstructing 3D mesh from video image sequences supervisor : Mgr. Martin Samuelčik by Martin Bujňák specifications Master thesis

Streams Reading: 2 nd Ed: , rd Ed: 11.1, 19.1, 19.4

The content of these slides by John Galeotti, © Carnegie Mellon University (CMU), was made possible in part by NIH NLM contract# HHSN P,

ITK Input/Output Kitware Inc.. Overview IO Factory Mechanism Image File IO Transform File IO SpatialObject File IO Logger.

1 Computer Graphics Week2 –Creating a Picture. Steps for creating a picture Creating a model Perform necessary transformation Lighting and rendering the.

NA-MIC National Alliance for Medical Image Computing Slicer and ITK Raul San Jose.

Wiimote/Kinect Lab Midterm Update Senior Design December 2011, Group 16 Adviser: Dr. Tom Daniels Brenton Hankins Rick Hanton Harsh Goel Jeff Kramer.

Spatiotemporal Saliency Map of a Video Sequence in FPGA hardware David Boland Acknowledgements: Professor Peter Cheung Mr Yang Liu.

File Systems cs550 Operating Systems David Monismith.

1 SICBDST and Brunel Migration status and plans. 2 Migration Step 1: SICBMC/SICBDST split  Last LHCb week: Split done but not tested  Software week.

Chapter 7 - Interprocess Communication Patterns

Daniel Blezek, Jesus Caban, Brad Lowkamp, Dan Muller, Fabrice de Chaumont, Julien Michel, Harvey Cline, Gabe Hart, Ghassam Hamarech, John Galeotti, Raghu.

1 ITK Lecture 6 - The Pipeline Methods in Image Analysis CMU Robotics Institute U. Pitt Bioengineering 2630 Spring Term, 2006 Methods in Image Analysis.

Final Year Project. Project Title Kalman Tracking For Image Processing Applications.

Quiz Week 8 Topical. Topical Quiz (Section 2) What is the difference between Computer Vision and Computer Graphics What is the difference between Computer.

Bill Hoffman, Jesus Caban, Brad Lowkamp, Dan Muller, Fabrice de Chaumont, Julien Michel, Harvey Cline, Gabe Hart, Ghassam Hamarech, John Galeotti, Raghu.

Application Design Document Developers: o Uri Goldenberg o Henry Abravanel o Academic.

I/O aspects for parallel event processing frameworks Workshop on Concurrency in the many-Cores Era Peter van Gemmeren (Argonne/ATLAS)

April / 2010 UFOCapture 1 UFOCaptureV2 Time-Shift-Motion-Detect-Video-Recording software for complete records of un-expected events.

Mini-Workshop on multi-core joint project Peter van Gemmeren (ANL) I/O challenges for HEP applications on multi-core processors An ATLAS Perspective.

OpenCV C++ Image Processing

Graphics Pipeline Bringing it all together. Implementation The goal of computer graphics is to take the data out of computer memory and put it up on the.

NA-MIC All Hands Meeting - 1/9/2006

UFOCaptureV2 Time-Shift-Motion-Detect-Video-Recording software for complete records of un-expected events April / 2010 UFOCapture.

Rotoscoping Senior Capstone Project | Ted Trisco

Kai Li, Allen D. Malony, Sameer Shende, Robert Bell

Hand Gestures Based Applications

Graphcut Textures:Image and Video Synthesis Using Graph Cuts

Depth Analysis With Stereo Cameras

Steve Pieper, PhD Raul San Jose, PhD Alex Yarmarkovich, PhD

Graphics Fundamentals

Core 1b – Engineering Data and Clinical Interface

Image Segmentation Classify pixels into groups having similar characteristics.

Vision for Robotic Applications

The Graphics Rendering Pipeline

TerraForm3D Plasma Works 3D Engine & USGS Terrain Modeler

Computer Architecture

SyDEVS Library Building7m Tutorial

Workshop II UU Crowd Simulation Framework

Dynamic Packet-filtering in High-speed Networks Using NetFPGAs

ITK Architecture Kitware Inc..

Virtual Memory Overcoming main memory size limitation

Chapter 1: Creating a Program.

Presentation transcript:

ITK Video ITK Extensions for Video Processing Amitha Perera, Patrick Reynolds, Matt Leotta, Luis Ibanez, Gabe Hart Real-Time Image Capture for ITK through a Video Grabber Kevin Cleary Methods in Medical Image Analysis: An ITK-Based Course with Deliverable Algorithms that extends and evaluates ITK while broadening its developer base John Galeotti

Goal Make video a first-class citizen of ITK Video I/O Allow ITK to be used with Computer Vision libraries (OpenCV, vxl, etc.) Allow other libraries to be used with ITK Implement ITK filters using other libraries Ensure framework allows real-world video problems to be researched E.g. don't try solve hard real-time, but make sure real-time issues can be solved

Assumptions and Choices Time is not another dimension Video != 3-d pixel volume Unbounded (live cameras) Memory (2 hrs 1920x1080 color video = 44 GB) On-line processing (streaming applications) We haven't invented time travel yet No explicit tagging of color spaces and other metadata (except for time) Video readers will produce grayscale or RGB Similar to CT and Hounsfield

Status: Real-time capture Using avacap for frame grabbing (GPL) Created an ITK version which is not GPL Ring buffer implemented (duplicated)

Status: Video Extensions 4-hr tutorial at CVPR Presented: ITK; using ITK with OpenCV; using OpenCV with ITK; direct video support in ITK ~33 people stayed for whole session Raffle ticket histogram: 1: 22 2: 6 3: 15 4: 5 5: 7 6: 4 7: 2

Video in ITK reader->SetFileName( argv[1] ); // video reader writer->SetFileName( argv[2] ); // video writer frameDifferenceFilter->SetFrameOffset(1); videoCaster->SetImageFilter( imageCaster ); imageThresh->ThresholdBelow( 128 ); videoThresh->SetImageFilter( imageThresh ); imageCurvatureFlowFilter->SetTimeStep( 0.5 ); imageCurvatureFlowFilter->SetNumberOfIterations( 20 ); videoCurvatureFlowFilter->SetImageFilter( imageFilter ); videoCurvatureFlowFilter->SetInput( reader- >GetOutput() ); videoCaster->SetInput( videoFilter->GetOutput() ); frameDifferenceFilter->SetInput( videoCaster- >GetOutput() ); videoThresh->SetInput( frameDifferenceFilter- >GetOutput() ); writer->SetInput( videoThresh->GetOutput() ); writer->Update(); // generate output video frame by frame, without reading whole source video at once

New Classes

New classes Real-time capture: A new video I/O classes wrapping a capture library Course: ITK video filters ITK representations of non-pixel data (tracks, etc.)

Class Hierarchy itk::DataObject itk::ProcessObject itk::TemporalRegion itk::RingBuffer itk::TemporalProcessObject itk::TemporalDataObject itk::VideoFileWriter itk::VideoIOBase itk::ImageIOBase itk::OpenCVVideoIO itk::VXLVideoIO itk::FileListVideoIO itk::VideoStream itk::VideoSource itk::VideoFileReader itk::VideoToVideoFilter itk::FrameDifferenceVideoFilter itk::FrameAverageVideoFilter itk::DecimateVideoFilter itk::ImageFilterToVideoFilterWrapper

Temporal Data Objects

itk::TemporalDataObject itk::TemporalRegion itk::TemporalDataObject FrameStart FrameDuration LargestPossibleTemporalRegion RequestedTemporalRegion RealStart BufferedTemporalRegion RealDuration DataObjectBuffer itk::RingBuffer 1 3 2 5 4 ... itk::DataObject

std::map< frame num, … > itk::VideoStream itk::TemporalRegion itk::VideoStream FrameStart FrameDuration LargestPossibleTemporalRegion RequestedTemporalRegion RealStart BufferedTemporalRegion RealDuration DataObjectBuffer itk::Region itk::RingBuffer LargestPossibleSpatialRegionCache LargestPossibleSpatialRegionCache itk::Point LargestPossibleSpatialRegionCache 1 std::map< frame num, … > itk::Vector 3 2 OriginCache itk::Matrix SpacingCache 5 4 ... DirectionCache itk::Image

Temporal Process Objects

itk::TemporalProcessObject itk::TemporalDataObject itk::TemporalDataObject itk::TemporalDataObject itk::TemporalDataObject Input Output n -1 n n +1 n n +1 UnitInputNumberOfFrames UnitOutputNumberOfFrames InputStencilCurrentFrameIndex FrameSkipPerOutput

itk::VideoToVideoFilter itk::TemporalDataObject itk::TemporalDataObject itk::VideoStream itk::TemporalDataObject Input Output n -1 n n +1 n n +1 UnitInputNumberOfFrames UnitOutputNumberOfFrames InputStencilCurrentFrameIndex FrameSkipPerOutput

itk::ImageToImageFilter itk::VideoToVideoFilter itk::ImageToImageFilter →GenerateData Process GenerateData() ThreadedGenerateData( … )

itk::VideoToVideoFilter itk::VideoToVideoFilter →GenerateData Process GenerateData() TemporalStreamingGenerateData() ThreadedGenerateData( … )

Pipeline Architecture

ITK Pipeline

UpdateOutputInformation ITK Pipeline UpdateOutputInformation Reader Filter A Filter B Update()

UpdateOutputInformation ITK Pipeline UpdateOutputInformation Reader Filter A Filter B Update()

UpdateOutputInformation ITK Pipeline UpdateOutputInformation Reader Filter A Filter B Update()

ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion Filter A Filter B Update()

ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion Filter A Filter B Update()

ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion UpdateOutputData Filter A Filter B Update()

ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion UpdateOutputData Filter A Filter B Update()

ITK Pipeline UpdateOutputInformation Reader PropagateRequestedRegion UpdateOutputData Filter A Filter B Update()

ITK Video Pipeline

UpdateOutputInformation ITK Video Pipeline UpdateOutputInformation 0:1 Reader 3:1 Filter A 2:1 Filter B Update()

UpdateOutputInformation ITK Video Pipeline UpdateOutputInformation 0:1 Reader 3:1 Filter A 2:1 Filter B Update()

UpdateOutputInformation ITK Video Pipeline UpdateOutputInformation 0:1 Reader 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

ITK Video Pipeline UpdateOutputInformation 0:1 Reader PropagateRequestedRegion UpdateOutputData 3:1 Filter A 2:1 Filter B Update()

OpenCV / VXL Bridges

Single Image Bridges itk::OpenCVImageBridge IplImage ↔ itk::Image cv::Mat ↔ itk::Image itk::VXLImageBridge (forthcoming) vil_image ↔ itk::Image vidl_frame ↔ itk::Image

Video Bridges itk::OpenCVVideoCapture Subclass of OpenCV's cv::VideoCapture Takes itk::VideoStream as input Triggers pipeline attached to input if necessary itk::vidl_itk_istream Subclass of VXL's vidl_istream

IO Mechanism

IO Mechanism Itk::VideoFileReader Itk::VXLVideoIO Itk::VXLVideoIOFactory Itk::VideoStream Itk::VideoIOFactory Itk::VideoIOBase Itk::ImageIOBase Itk::OpenCVVideoIOFactory Itk::OpenCVVideoIO Itk::VideoFileWriter

Module Structure

Module Structure - Current VideoCore VideoFilters VideoIO Common OpenCV VXL Common OpenCV VXL Core OpenCV VXL FileList Directories Modules / Groups Video-Core-Common Video-Filters-Common Video-IO-Core Video-Core-OpenCV Video-Filters-OpenCV Video-IO-OpenCV Video-Core-VXL Video-Filters-VXL Video-IO-VXL Video-IO-FileList Group_VideoCommon Group_VideoOpenCV Group_VideoVXL

Module Structure – Desired (?) VideoCore VideoFilters VideoIO Common OpenCV VXL Common OpenCV VXL Core OpenCV VXL FileList ? ? ? Directories Modules / Groups Video-Core-Common Video-Filters-Common Video-IO-Core Video-Core-OpenCV Video-Filters-OpenCV Video-IO-OpenCV Video-Core-VXL Video-Filters-VXL Video-IO-VXL Video-IO-FileList Group_VideoCommon Group_VideoOpenCV Group_VideoVXL

Issues/ToDo RealTimeStamp is not a real time stamp I/O readers Image sequence as a video (done via file list) Multi-frame DICOM file as a video Video capture sync with Children's National on video reader and on using framework Direct video I/O (using ffmpeg) VideoStream → 3-d volume converter Handle infinite duration Implement timestamp-based flow (vs frame-based flow) Implement video filter base classes for multiple inputs and outputs Synchronization of multiple streams Avoid memcpy where possible (when wrapping other libraries) Ensure frame size is not an intrinsic problem Working with SD video now, but will need to work with HD video Implement filters in new video framework (e.g. using OpenCV) Translators for other data structures (e.g. OpenCV sequences, tracks, etc.)

END