1. E. Borovikov, A. Sussman, L. Davis, University of Maryland
An Efficient System for Multi-Perspective Imaging and Volumetric Shape Analysis
We present a high performance system for efficient multi-perspective image analysis on very large image datasets, implemented as a customized extension of the Active Data Repository (ADR) object-oriented framework.
The resulting system provides a flexible framework for handling multi-perspective video sequences in any parallel or distributed computing environment that can support ADR. We have employed
the framework to produce an efficient volumetric shape analysis application implementation. Initial performance results show that using an effective data distribution algorithm to produce good load balancing allows the ADR implementation to achieve scalable high performance.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

2. High-performance Multi-perspective Vision Studio
Multi-view vision apps made user friendly
Organize multi-view video data
Efficient back-end processing & user-friendly front-end
Portability across computing platforms
Parallel computers
Distributed systems
Independence from data acquisition facilities
Various sensor configurations, e.g. camera positioning
Images: color or b/w, at various resolutions
Extensibility
R&D friendly
A multi-perspective visual studio. Why anybody would care to build a system like that?
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

3. Multi-perspective Vision Studio
DAF
DAF
DAF
Separation from data acquisition facilities
Generic multi-view sequence management
Extensible multi-perspective vision application framework
Loader
Database
Front-end services
Back-end services
App
App
App
App
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

4. Multi-view Video Sequence Data
Acquisition: the Keck Lab
64 cameras
85 frames/sec
1 min = 95GB
Distributed system
PC cluster
Storage 9TB
Parallel processing
Active Data Repository (ADR)
Multiple video streams generate very large amounts of image data that can become unmanageable
unless there is an efficient way to store, catalog and process them.
The Keck Lab at the University of Maryland consists of 64 cameras synchronously capturing synchronous video streams at a rate of up to 85 frames per second, with each frame being 644  484  1 bytes. One minute of such multi-perspective video requires about 95 GBytes of storage.
We describe how to customize Active Data Repository (ADR) for building an efficient and flexible framework for storing multi-perspective image data and performing visual analysis. Further, we describe a volumetric shape recovery and analysis application that is based on this framework, and give some general guidelines
on developing multi-perspective imaging applications using the system.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

5. ADR-based Imaging System
Framework for multi-perspective imaging
ADR application structure
Customizing ADR
framework:
back-end
ADR front-end
user:
app front-end
clients
With a number of other multi-perspective imaging applications in mind, we have built a relatively generic multi-perspective imaging framework based on ADR.
ADR is an object-oriented framework designed to efficiently integrate application-specific processing with the
storage and retrieval of multi-dimensional datasets on a parallel machine or cluster of workstations with a disk farm.
The image analysis framework discussed in this paper customizes ADR’s parallel back-end (PBE) and front-end (FE). The implementation also provides a tool for loading and cataloging multi-perspective sequences of image data. This leaves the image analysis application developer with the task of providing an application-specific front-end and a client.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

6. Multi-perspective Imaging Framework
Multi-perspective image data
Data set: multi-perspective video sequence
Data chunk: single image <camera, time>
Operations: associative and commutative
Parallel back-end (PBE)
Storage, retrieval, projection, aggregation
Camera index lookup
Application and ADR front-end (FE)
Maintain meta-information about datasets
Translate queries from clients to PBE
multi-perspective image data
A multi-perspective video sequence constitutes a single image dataset.
A data element (chunk) is a single image whose attributes are <camera index, time index>.
To populate the database, a user employs the data loader tool supplied with the framework.
Operations on data elements have to be associative and commutative.
PBE
The back-end is responsible for storing the datasets and carrying out the requested input data retrieval, projection to the output space, and aggregation operations.
The default index service maps a four-dimensional query region (three for space, one for time) into a set of relevant chunks. This is accomplished via a camera index lookup computed for each query region as needed.
The default projection maps the <camera-index, time-index> pair to the <time-index>, to allow aggregation across all cameras that view a spatial region at a given time.
No default aggregation operation is supplied, since aggregation is application dependent.
Any default services can be customized by the application developer. FE
The application front-end interacts with clients via an application-specific communication protocol, and translates queries into a common format for the ADR front-end, which will send the reformulated query to the PBE.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

7. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Volume
Recovery
App
multi-perspective
silhouette-based
visual cone intersection
octree encoding
The application uses multi-perspective image data to reconstruct the volumetric shape of a foreground object. The application then can render the volume from an arbitrary vantage point at any point in time and allow users to analyze the 3D shape by requesting region-varying resolution in the reconstruction.
The volumetric reconstruction is based on the distributed visual cone intersection technique.
The reconstructed volume is represented as an occupancy map encoded by an octree, encoded as a byte array in DFS-order for network transmission efficiency
A database inquiry involves specifying a 3D region, a time range and a volumetric reconstruction resolution.
The query result is a multi-resolution reconstruction of the fore-ground object region lying within the query region. This information can be used for 3D shape analysis and 3D tracking.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

8. Multiview Visual Cone Intersection
For each view
extract object’s silhouettes
build visual cones
Global combine
collect occupancy maps
intersect them
Starting with the image of the 3D object, for each view
compute the silhouette
build object’s visual cone as a 3D occupancy map
represent visual cone as an octree
intersect octant’s projection bounding square with object’s silhouette to decide octant’s occupancy
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

9. Multiview Silhouette Extraction- = - =
Estimate the foreground object's volume
multi-perspective silhouette segmentation
silhouette visual cone intersection
volumetric data streaming
graphical rendering and analysis often are done off-line (time consuming) - =
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

10. Depth of Reconstruction
volumetric shape reconstructed to the user specified octree depth
depth of reconstruction is region-specific
reconstruction depth is finite due to discrete images
SOW THE MULTIPERSPECTIVE MOVIE HERE
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

11. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Experiments
Workload balancing
data distribution strategies
query scenarios
Scaling
speedup
scaled speedup
Experimented with:
Workload balancing
data distribution across the disk farm
query scenarios
Scaling
n vs 2n processors same workload
x job on n processors 2x job on 2n processors
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

12. E. Borovikov, A. Sussman, L. Davis, University of Maryland
System Setup
Back-end: Linux PC cluster
16 nodes
450MHz dual-processor Pentium II
256 MB RAM
Gigabit Ethernet
Front-end: Pentium II PC, same network
Client: 400 MHz dual-processor Pentium II PC, 100Mbit/s TCP/IP network
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

13. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Test Queries
Fixed
query universe: 2  2  2 [m] cube
camera calibration parameters
Varying
# of frames: 1, 50, 100, 200, 400
# of processors: 1, 2, 4, 8, 16
depth of reconstruction: from 0 to 12
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

14. Data Distribution Strategies
view per processor
round robin in a single dimension
view per processor
round robin in a single dimension
random
Hilbert space-filling curves
declusters data randomly
requires good n-D pseudo random generator
has a flavor of randomness, but
distributes data periodically,
worst case occurs when
dim_max_index mod processors_count = 0
naïve
works OK when # of views = # of processors
poor workload balance otherwise
practical approach to multidimensional data distribution
robust, inexpensive declustering of compact regions
Can’t use first two in general case
Random declustering is OK, but it’s random; can do better utilizing locality of queries
Hilbert space-filling curve method:
generate the curve indexes
sort them
distribute as round robin across the nodes
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

15. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Workload Experiments
Frame distribution
Random
Hilbert curve
nodes
frames
mean
std div 4
0:4:399
325
15.72
37.99
0:2:399
650
12.08
37.57
0:1:399
1300
24.25
0.00 8
162
18.66
34.81
18.11
44.19
17.09 16 81
10.63
46.74
17.00
30.89
13.56
Frame hit count statistics:
Random chunk declustering results in lower standard deviations in all but whole space queries; in which case the Hilbert curve based declustering method results in a perfectly even distribution.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

16. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Scaling Experiments
Random
# of frames
100
200
400
# prc 4
148
297
592 8
104
211
414 16 79
160
320
Query turnaround times (sec)
Hilb crv
# of frames
100
200
400
# prc 4
135
271
539 8 90
176
349 16 79
152
305
Query turn around times in seconds.
The Hilbert curve based declustering results in slightly better overall performance and better speedups due to spreading chunks that near each other (in the input attr space) across different nodes.
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland

17. E. Borovikov, A. Sussman, L. Davis, University of Maryland
Summary
Goals
multi-perspective vision studio
separation between sensor net and vision application
Customized ADR for multi-perspective imaging
portability across parallel platforms
robustness in handling large datasets
Built multi-perspective imaging framework
good workload balance using Hilbert curve declustering
robust scaling
Utilized it for a volumetric shape analysis application
variable time step size volumetric sequences
variable space resolution
10/13/2018
E. Borovikov, A. Sussman, L. Davis, University of Maryland