1. Nonparametric Hypothesis Tests for Dependency Structures
John Fisher
MIT CSAIL/LIDS
Alexander Ihler, Alan Willsky

2. Do these depend on each other (and how)?

3. Motivation/Problem Domain
Heterogenous sensors.
E/O, I/R, acoustic, MRI, fMRI, etc.
Need to perform local fusion to support global inference.
How do we aggregate localized measurements of distributed phenomenon?
How do we do principled statistical inference when the model is only partially specified?
Critical need to understand statistical relationships between sensor outputs in the face of many modes of uncertainty (sensors, scene, geometry, etc).

4. Observations
When asking simple questions we can use simple/approximate models.
Performance and data fusion complexity is dictated by the complexity of the hypotheses.
Many problems involving data fusion can be cast as hypotheses between graphical models.
In such cases the problem decomposes into terms related to statistical dependency versus modeling assumptions

5. AV Association at the Signal Level
Sounds and motions which are consistent can be attributed to a common cause
Question: How do we quantify consistent?
consistent
inconsistent

6. Data Association as a Hypothesis Test
Information theoretic quantities such as mutual information, Kullback-Leibler divergence arise naturally in the context hypothesis testing
vs.
0.61

7. A/V Association as a Hypothesis Test
Assuming independent sources, hypotheses are of the form
Asymptotic comparison of known models to those estimated from a single realization

8. Asymptotics of Likelihood Ratio
Decomposes into two sets of terms:
Statistical dependencies (groupings)
Differences in model parameterizations

9. Asymptotics of Likelihood Ratio
If we estimate from a single realization:
Statistical dependence terms remain
Model divergences go away

10. High Dimensional Data
Learn low-dimensional auxiliary variables which summarize statistical dependency of measurements

11. AV Association/Correspondence
association matrix for 8 subjects
0.68
0.61
Table contains MI (equivalently Likelihood) for each possible association
Hypothesis over any two possible associations is difference of terms in table
0.19
0.20

12. AV Association/Correspondence
association matrix for 8 subjects
0.68
0.61
Hungarian algorithm (or Auction algorithms) provides efficient means for evaluating all possible associations.
MI scores are the natural statistic to fill the table with.
Note that the complexity in filling the table is N^2 in the number of sources.
0.19
0.20

13. General Structure Tests
Generalization to hypothesis tests over graphical structures
How are observations related to each other? vs vs

14. General Structure Tests
Intersection Sets - groupings on which the hypotheses agree H1 vs H2
Nominal set algebra
-most sets are empty
-at most, D (the number of variables) are non-empty
-pigeon-hole, variables can appear in at most one set – complexity is order D^2

15. General Structure Tests
Asymptotics have a similar decomposition as in the 2-variable case (via the intersection sets):

16. General Structure Tests
Extension of previous description data association is straightforward for such tests.
Estimation from a single realization incurs a reduction in separability only in terms of the model difference terms.
The “curse of dimensionality” (with respect to density estimation) arises in 2 ways:
Individual measurements may be of high dimension
Could still design low dimensional auxiliary variables
The number of variables in a group
New results provide a solution

17. General Structure Tests
The test implies potentially 6 joint densities, but is simplified by looking at the intersection sets. H1 H2

18. General Structure Tests
High dimensional variables
learning auxiliary variables reduces dimensionality in one aspect.
But we would still have to estimate a 3 dimensional density.
This only gets worse with larger groupings.

19. K-L Divergence with Permutations
Simple idea which mitigates many of the dimensionality issues.
Exploits the fact that the structures are distinguished by their groupings of variables.
Key Ideas:
Permuting sample order between groupings maintains the statistical dependency structure.
D(X||Y) >= D(f(X)||f(Y))
This has the advantage that we can design a single (possibly vector-valued) function of all variables rather than one function for each variable.

20. K-L Divergence with Permutationsf

21. High-Dimensional, Multi-modal Data Association
(a) grayscale
(c) hue
Top row is what the “sensor” sees. Bottom row is original image to make it easier for audience to see which image goes with which.
(b) grayscale
(d) img diff

22. High-Dimensional, Multi-modal Data Association
(a) grayscale
(c) hue
Top row is what the “sensor” sees. Bottom row is original image to make it easier for audience to see which image goes with which.
(b) grayscale
(d) img diff

23. Data Association Example
LLR estimates for pair-wise associations (left)
Compared to the distribution over the null hypothesis
Distribution of full association (middle)
Incorrect association likelihood shows some global scene dependence (e.g. due to common lighting changes)

24. More General Structures
Analysis has been extended to comparisons between triangulated graphs.
Can be expressed as sums and differences of product terms.
Admits a wide class of Markov processes.

25. Modeling Group Interactions
Object 3 tries to interpose itself between objects 1 and 2.
The graph describes the state (position) dependency structure.

26. Modeling Group Interactions

27. Association vs Generative Models
One instantiation of IT fusion approach is equivalent to learning a latent variable model of the audio video measurements.
Random variables:
Parameters, appearance bases:
Simultaneously learn statistics of joint audio/video variables and parameters
as the statistic of association (consistent with the theory)

28. Incorporating Nuisance Parameters
Extension of multi-modal fusion to include nuisance parameters
Audio is an indirect pointer to the object of interest.
Combine motion model (nuisance parameters) with audio-video appearance model.

29. Incorporating Motion Parameters
without motion
model
example frames
average image
with motion
model

30. High-Dimensional, Multi-modal Data Association
grayscale
hue
grayscale
img diff
Top row is what the “sensor” sees. Bottom row is original image to make it easier for audience to see which image goes with which.
(a)
(c)
(b)
(d)

31. High-Dimensional, Multi-modal Data Association
grayscale
hue
grayscale
img diff
Top row is what the “sensor” sees. Bottom row is original image to make it easier for audience to see which image goes with which.
(a)
(c)
(b)
(d)