1. Estimating Networks With Jumps
Mladen Kolar ( )
Carnegie Mellon University
With Eric P. Xing

2. The Internet
[wikipedia]

3. Twitter

4. Biological regulatory network

5. Metabolic network
In this image we show the integrated metabolic and originated physical enzyme-enzyme interaction network of the Mycobacterium tuberculosis. We created them separately and then integrated it into a single network. Nodes are the enzymes catalyzing reactions, the red directed edges are the metabolic pathway and the blue undirected edges are predicted physical interactions.
[Daniel Banky, ISMB09]

6. Networks are ubiquitous in sciences
… and nowadays worldwide

7. Networks are mathematical abstractions of complex systems
Networks are useful for visualization discovery of regularity patterns exploratory analysis ... of complex systems.

8. Current practices in network modeling
Probabilistic graphical models used for exploring networks
Ising models
Gaussian graphical models
contains both structure and parameters

9. Interpretation of Markov Random Fields
A network can be obtained by drawing links between nodes that are conditionally independent.

10. High dimensional inference
Applications in many domains have large number of variables and small number of observations
To avoid curse of dimensionality the models are assumed to have a low-dimensional structure
for example, a small number of non-zero parameters – sparsity of the precision matrix

11. Rich literature on estimation of MRFs
Yuan & Lin, 2006; Meinshausen and Buhlmann, 2006; d’Aspremont et al., 2007; Bickel & Levina, 2007; El Karoui, 2007; Rothman et al., 2007; Zhou et al., 2007; Friedman et al., 2008; Lam & Fan, 2008; Ravikumar et al., 2008; Zhou, Cai & Huang, 2009; Peng et al. 2009; Guo et al., 2010; …

12. Estimating sparse Gaussian MRF models
Penalized maximum likelihood approach Neighborhood selection

13. Relating Neighborhood Selection to Linear regression
Partial correlation represents correlation between two variables conditioned on the rest

14. Graph Regression
Neighborhood selection

15. Graph Regression

16. Graph Regression

17. Drawbacks of current approaches
Many systems of interest cannot be explained by one static network model. There is a need for models that explain dynamical systems.

18. Estimating Time-Varying Networks

19. Networks with jumps

20. Temporally Smoothed Graph Regression (TESLA)
[Ahmed and Xing, 2009] …

21. Improved Estimation Procedure
Estimation of neighborhood of a node
Loss
Penalty

22. The structure of the penalty
Structural changes
Sparsity

23. Optimization Convex problem
Non-smooth penalty term presents difficulties
Smoothing technique (Nesterov, 2005)
Smooth approximation of

24. Optimization (II) Accelerated gradient applied to the smoothed problem
iteratively solves
It can be shown that the algorithm converges as

25. Tuning parameter selection
Optimizing the Bayesian information criterion

26. Simulation results (Chain)

27. Simulation results (NN)

28. Definition of the model
The model where defines a block with being block boundaries

29. Assumptions
A1 There exist two constants and such that A2 Variables are scaled so that

30. Assumptions (II)
A3 There exists a constant A4 There exists a constant

31. Assumptions (III)
A5 The sequence of partition boundaries satisfy , where is a fixed, unknown sequence of the boundary fractions belonging to [0, 1].

32. Consistent estimation of fraction boundaries
Under A1 – A5, with the number of blocks known, we can show that
with for some , if the following holds
Here the minimal size of the jump measured as

33. Proof strategy
The proof hinges on the analysis of the optimality conditions where and We show that events occur with low probability, otherwise the optimality conditions cannot be satisfied.

34. Consistent estimation of fraction boundaries (II)
Under the same regularity conditions, with an upper bound on the number of blocks known
The distance h(., .) is defined as

35. Structural consistency
We have shown that the partition boundaries can be estimated consistently. Under the same regularity condition we can further show that for blocks that have “enough” samples

36. Proof strategy
The proof uses techniques developed in Meinshausen and Buhlmann, 2006; Peng et al and Wainwright 2009 The main difficulty that differs from the existing work is controlling the bias that arises from estimating the partition boundaries.

37. Discussion
Confidence on the estimated networks Stability Applications to streaming data and online learning

38. Thank you!