1. VideoForge An image processing system

2. Contents Current state of the art IntroductionTiers Base Base Core Core Modules Modules User interface Interactive mode Interactive mode Scripting Scripting universal interface primary implementation COM scripting object Implementation templates Controllers Controllers Algorithms Algorithms

3. Current state of the art Libraries Loki, Boost Loki, Boost Direct Show™, ACML, IPL(IPP), OpenCV, Microsoft Vision SDK, ImageMagick, … Direct Show™, ACML, IPL(IPP), OpenCV, Microsoft Vision SDK, ImageMagick, …Applications GraphEdit (part of the DirectShow) GraphEdit (part of the DirectShow) Photoshop Photoshop ImageJ ImageJ ImageForge ImageForge Adobe Premiere, … Adobe Premiere, …

4. Introduction System for video and live-stream image processing Platform independent ANSI C++ ANSI C++ Easy-to-do computer vision Three tier application Base (Serialization, Communication, Logging) Base (Serialization, Communication, Logging) Core (Module management, Memory management) Core (Module management, Memory management) Modules (Devices, Algorithms,) Modules (Devices, Algorithms,)

5. Base Serialization Binary vs. textual storage Binary vs. textual storage Automatic vs. manual storage Automatic vs. manual storage Type lists, storing of arrays, classes … Type lists, storing of arrays, classes … Variable reflection offers the possibility of calling the variable “by name” offers the possibility of calling the variable “by name” m_plugin.FindVar (“varName”); m_plugin.FindVar (“varName”); m_plugin.GetVarNames(…); m_plugin.GetVarNames(…);

6. User interface Communication Scripting possibility Scripting possibility User interface (Interactive mode) User interface (Interactive mode) Types of communication Actions – non-parametric methods Actions – non-parametric methods Properties – like variables, accessor methods Properties – like variables, accessor methods Methods – methods with parameters Methods – methods with parameters Events – (e.g. onClose, …) Events – (e.g. onClose, …)

7. Core Module and plug-in management Enumeration Enumeration Registration Registration Memory management Memory (de-)allocation Memory (de-)allocation Correct memory handling for erroneous modules Correct memory handling for erroneous modules Actions (from plug-ins and core itself) User interface handling User interface handling

8. Modules and plug-ins Module – file Contains plug-in implementations Contains plug-in implementations compiled-in or represented by DLL compiled-in or represented by DLLPlug-ins Sources Sources Feeds data into chain Algorithms Algorithms Transforms data, somehow Special plug-ins Special plug-ins usually compiled : persistence, etc.

9. Image & Frame similar to IPL IMAGE structure contains: resolution resolution ROI (rectangle of interest) ROI (rectangle of interest) MediaType MediaType color mode (rgb, YUV, etc.) bpp (=color depth) Frame time information (start, stop time, fps) time information (start, stop time, fps) previous & next frame reference (=pointer) previous & next frame reference (=pointer)

10. Building the graph Pipeline building Interactive mode vs. non-interactive mode Interactive mode vs. non-interactive mode Device (Controller = Device enumerator) Algorithms (Algorithm chain = Composite algs)

11. System windows Windows show graphical information show graphical information part of a particular plugin part of a particular plugin Bitmap windows shows contents of an image buffer shows contents of an image buffer Vector windows usually used to show some information in the image usually used to show some information in the image applied as a second layer onto the bitmap window applied as a second layer onto the bitmap window Window events OnClose, … OnClose, …

12. Templates (1/3) Application template UI application UI application Console application Console application Module template default implementation default implementation rare modification rare modification implements a list of plug-ins implements a list of plug-ins

13. Templates (2/3) Controller template Implement source enumeration Implement source enumeration Implement data feed Implement data feed Possibly implement UI Possibly implement UI Algorithm template Implement data transformation Implement data transformation Possibly implement UI Possibly implement UI User must implement 1 method User must implement 1 method

14. Conclusion We proposed system for implementation of image processing and computer vision tasks as platform-independent as possible (ANSI C++) as platform-independent as possible (ANSI C++) consists of (lightweight) core and plugins consists of (lightweight) core and plugins offers a user interactive, scriptable user interface offers a user interactive, scriptable user interface offers various templates for fast implementation offers various templates for fast implementation The system is a part of various grants Grant UK Grant UK APVV Grant APVV Grant

15. Future work Finalize implementation of the core with the basic modules (12/05) basic image processing operations basic image processing operations Implementation of basic modules of 0(1/06) working set optimization working set optimization Implementation of special modules (06/06) Feature tracking Feature tracking Cooperation with avatars Cooperation with avatars Statistics – classifier training Statistics – classifier training