1. Application Framework for Computer Vision
The PALLAS Group

2. The reason we are here A programming environment that simultaneously:
enables highly efficient implementation on parallel processors
allows natural expression of computer vision algorithms
What do these programs look like?
Hypothesis: They are mostly composed from a collection of pre-existing algorithms

3. Hypothesis: Assembly of Parts
Computer Vision systems are assembled from a set of pre-existing components
Vision and Learning communities have produced a large toolbox, along with standard mathematical tools.
Even when part of a system is entirely novel
Nothing can stand in complete isolation
How often is an algorithm developed, deemed useful, and then never used again?
When it is used again, how similar is the implementation?

4. Example: Recognition via Poselets
Gradients
Sgemm
MS-Vector
Distances
Image
Histogram
ThresholdReduction
Update
Merge Clusters
HOG Feature Extraction
Linear SVM Classifiers
Mean-Shift
Clustering
Agglomerative
Clustering
Every component existed previously. Novelty is in the composition.

5. Example: gPb Contour Detector
Image
Intervening Contour
Generalized Eigensolver
Convert Colorspace
Form Eigenvectors
SpMV
SGEMM
Filter bank
Convolution
Integral Images
BLAS1
BLAS1
BLAS1
BLAS1
BLAS1
K-means
Gradients
Filter bank
Convolution
Histogram
Show this before circus parade of components?
SGEMM
Filter bank
Convolution
Combine, Normalize
Relabel
Combine
Skeletonize
Non-max
suppression
Contours

6. Bag-of-Words Object Recognition
Harris-Affine Regions
Per-Class
Histograms
Training Set
SIFT Descriptors
Codebook
Naive Bayes
K-Means Clustering
Nearest Neighbor
Harris-Affine Regions
Class Label
Test Image
SIFT Descriptors

7. What Components? Feature Detection / Extraction: Clustering:
Must Have: SIFT, SURF, HOG, Pb, MSER
Nice to have: Geometric Blur, Shape context, Harris-Laplace, Harris-Affine, ...
Clustering:
Spectral, Mean-Shift, Agglomerative, K-means, GMM
Classification:
Must Have: Nearest-Neighbor, Naive Bayes, Logistic Regression, SVM, HMM
Nice to Have: Gaussian Process, Decision Trees, RVMs, AdaBoost, ...
Distance Functions / Kernels:
Manhattan, Euclidean, Mahalanobis, L-infinity
Polynomial, RBF, Chi^2,

8. Definitions
A pattern is a solution to a recurring problem described in such a way that the solution may be uniquely applied to a wide variety of related problems.
An software architecture is a hierarchical composition of patterns describing the implementation of a piece of software.
A framework is a software environment in which users customization may only be in harmony with the underlying architecture.

9. More Definitions
Library: A software implementation of a computational pattern (e.g. BLAS) or a particular sub-problem (e.g. matrix multiply)
Domain specific language: A programming language (e.g. Matlab) that provides language constructs that particularly support a particular application domain. The language may also supply library support for common computations in that domain (e.g. BLAS). If the language is restricted to maintain fidelity to a structure and provides library support for common computations then it encompasses a framework (e.g. NPClick). The language necessarily has Turing completeness /computes partial recursive functions.

10. What is the environment?
Proposal 1: A Mex-Library of fast, parallelized components
Proposal 2: Fixed-Architecture Frameworks with pre-fab parallelized components

11. Proposal 1: Mex Functions
Status Quo: We parallelize functions a la carte
Very difficult to modify: X,000 lines of Cuda code
It is very fast, but only as flexible as a C++ library
Can still be integrated into Matlab scripts: only the pre-parallelized functionality is fast.

12. Fixed-Architecture Frameworks
Example: Extract-classify framework
Customizable only via checkboxes and filling in callback functions
Includes library of fast, parallel features and classifiers
Useful primarily for application development, and building systems to test the algorithms you develop
Potential for inter-block parallelization
Training Set
Diff. of Gaussian
Training Features
Det. of Hessian.
MSER
SURF
SIFT
Geometric Blur
Logistic Regression
Test Image
Naive Bayes
Object Label
Quantize
Linear SVM
Feature Extraction
Classification

13. The Challenge You want to implement fundamentally new approaches
We can't provide a library that implements your next algorithm
Neither of the "Frameworks" nor the "Scripting Language" approach to composition allows a Vision researcher to produce a parallel implementation of a new component
E.G. Max-Margin Hough Transform is brand new. Multi-Month delay from idea's inception to parallel implementation, with us in the loop

14. Domain-Specific Language
Fine-grained components are integrated as intrinsics, accessed via function-call syntax
Some intrinsics (e.g. conjugate gradient) are library calls
Others guide implementation and optimization
Particular nesting and computation of components determines decisions made by a code generator
Too vague: how precisely is composition performed? Type-checking?
Run-time support?
Libraries invoked strictly by intrinsics????

15. Scripts vs Framework vs DSL
Framework enables assembling an application from pre-implmented parts
Provides access to a fast library of commonly used Vision and Learning algorithms
Potentially "easy" inter-block parallelization
Scripting Language also allows assembly from parts
Provides no ease of inter-block parallelization
Still provides access to fast libraries
DSL potentially allows implementation of new functionality by composing fine-grained components
Code Generator / JIT produces a parallel implementation
Enables extensibility of the Framework
Framework can provide guarantees about execution safety

16. The End

17. What is the environment?
Proposal 1: A Mex-Library of fast, parallelized components
Proposal 2: Fixed-Architecture Frameworks with pre-fab parallelized components
Proposal 3: A Domain-Specific Language with intrinsic-level integration of components