1. Modeling Real Graphs using Kronecker Multiplication
Jure Leskovec, Christos Faloutsos
Machine Learning Department

2. Modeling large networks
Large networks (e.g., web, internet, on-line social networks) with millions of nodes
Need statistical methods and models to quantify large networks

3. Some statistical property, e.g., degree distribution
The problem
We want to generate realistic networks
What are the relevant properties?
What is a good analytically tractable model?
How can we fit the model (estimate parameters)?
Given a large
real network
Generate a
synthetic network
Some statistical property, e.g., degree distribution
this talk

4. Why is this important? Gives insight into the graph formation process
Anomaly detection – abnormal behavior, evolution
Predictions – predicting future from the past
Simulations of new algorithms where real graphs are hard/impossible to collect
Graph sampling – many real world graphs are too large to deal with
“What if” scenarios

5. Statistical properties of networks
Features that are common to networks of different types:
Small-world effect [Milgram, Watts&Strogatz]
Degree distributions [Faloutsos et al]
Spectral properties [Chakrabarti et al]
Transitivity or clustering [Watts&Strogatz]
Community structure [Girvan&Newman, and others]
These properties are shared across many real world networks:
World wide web [Barabasi]
On-line communities [Holme, Edling, Liljeros]
Who call whom telephone networks [Cortes]
Internet backbone – routers [Faloutsos et al] …

6. Small-world effect Distribution of shortest path lengths
Distances in MSN messenger network
Distance (Hops)
log Number of nodes
Pick a random node, count how many nodes are at distance 1,2,3... hops
Distribution of shortest path lengths
Microsoft Messenger network
180 million people
1.3 billion edges
Edge if two people exchanged at least one message in one month period 7

7. Heavy-tailed degree distributions
Degree distribution of a blog network
Let pk denote a number (fraction) of nodes with degree k
We can plot a histogram of pk vs. k
Degrees in real networks are heavily skewed to the right
Distribution has a long tail of values that are far above the mean
Power law:
log(pk)
log(k)

8. Eigenvalue distribution in online social network
Spectral properties
Eigenvalue distribution in online social network
Eigenvalues of graph adjacency matrix follow a power law
Network values (components of principal eigenvector) also follow a power-law
log Eigenvalue
log Rank

9. Models of graph generation
Given graph properties
How can we design generative models that explain them?
Lots of work:
Random graph [Erdos and Renyi, 60s]
Preferential Attachment [Albert and Barabasi, 1999]
Copying model [Kleinberg et al, 1999]
Forest Fire model [Leskovec et al, 2005]
But all of these:
Do not obey all the properties (aim to model (explain) just one of the properties at a time)
Or are analytically intractable

10. The model: Kronecker graphs
Kronecker graphs are analytically tractable
We prove [with Chakrabarti, Kleinberg Kleinberg, Faloutsos in PKDD’05] that Kronecker graphs have rich properties:
Static Patterns
Power Law Degree Distribution
Small Diameter
Power Law Eigenvalue and Eigenvector Distribution
Temporal Patterns
Densification Power Law
Shrinking/Constant Diameter

11. Idea: Recursive graph generation
Intuition: self-similarity leads to power-laws
Try to mimic recursive graph / community growth
There are many obvious (but wrong) ways:
Kronecker Product is a way of generating self-similar matrices
Initial graph
Recursive expansion

12. Kronecker product: Graph
Intermediate stage
(3x3)
(9x9)
Adjacency matrix
Adjacency matrix

13. Kronecker product: Definition
The Kronecker product of matrices A and B is given by
We define a Kronecker product of two graphs as a Kronecker product of their adjacency matrices
N x M
K x L
N*K x M*L

14. Kronecker graphs We create the self-similar graphs recursively
Start with a initiator graph G1 on N1 nodes and E1 edges
The recursion will then product larger graphs G2, G3, …Gk on N1k nodes
We obtain a growing sequence of graphs by iterating the Kronecker product

15. Kronecker product: Graph
Continuing multypling with G1 we obtain G4 and so on …
G4 adjacency matrix

16. Stochastic Kronecker graphs
Create N1N1 probability matrix Θ1
Compute the kth Kronecker power Θk
For each entry puv of Θk include an edge (u,v) with probability puv
Probability of edge puv
Kronecker
multiplication
0.25
0.10
0.04
0.05
0.15
0.02
0.06
0.01
0.03
0.09
0.5
0.2
0.1
0.3
Instance
matrix K2 Θ1
For each puv flip Bernoulli coin
Θ2=Θ1Θ1

17. Kronecker graphs: Intuition
1) Recursive growth of graph communities
Nodes get expanded to micro communities
Nodes in sub-community link among themselves and to nodes from different communities
2) Node attribute representation
Nodes are described by features
[likes ice cream, likes chocolate]
u=[1,0], v=[1, 1]
Parameter matrix gives the linking probability
p(u,v) = 0.5 * 0.1 = 0.05
Little graph, super-graph (g1, g2) 1
0.5
0.2
0.1
0.3 Θ1

18. Properties of Kronecker graphs
We prove that Kronecker multiplication generates graphs that obey [PKDD’05]
Properties of static networks
Power Law Degree Distribution
Power Law eigenvalue and eigenvector distribution
Small Diameter
Properties of dynamic networks
Densification Power Law
Shrinking/Stabilizing Diameter
Good news: Kronecker graphs have the necessary expressive power
But: How do we choose the parameters to match all of these at once?     

19. Model estimation: approach
Maximum likelihood estimation
Given real graph G
Estimate Kronecker initiator graph Θ (e.g., ) which
We need to (efficiently) calculate
And maximize over Θ (e.g., using gradient descent)

20. Fitting Kronecker graphs
Given a graph G and Kronecker matrix Θ we calculate probability that Θ generated G P(G|Θ)
0.25
0.10
0.04
0.05
0.15
0.02
0.06
0.01
0.03
0.09 1
0.5
0.2
0.1
0.3 Θ Θk G
P(G|Θ)

21. Challenge 1: Node correspondenceΘk Θ
Nodes are unlabeled
Graphs G’ and G” should have the same probability
P(G’|Θ) = P(G”|Θ)
One needs to consider all node correspondences σ
All correspondences are a priori equally likely
There are O(N!) correspondences
0.25
0.10
0.04
0.05
0.15
0.02
0.06
0.01
0.03
0.09
0.5
0.2
0.1
0.3 σ G’ 1 1 3 2 4 G” 2 1 4 1 3
P(G’|Θ) = P(G”|Θ)

22. Challenge 2: calculating P(G|Θ,σ)
Assume we solved the correspondence problem
Calculating
Takes O(N2) time
Infeasible for large graphs (N ~ 105)
σ… node labeling
0.25
0.10
0.04
0.05
0.15
0.02
0.06
0.01
0.03
0.09 1 σ G
Θkc
P(G|Θ, σ)

23. Model estimation: solution
Naïvely estimating the Kronecker initiator takes O(N!N2) time:
N! for graph isomorphism
Metropolis sampling: N!  (big) const
N2 for traversing the graph adjacency matrix
Properties of Kronecker product and sparsity (E << N2): N2 E
We can estimate the parameters of Kronecker graph in linear time O(E)

24. Solution 1: Node correspondence
Log-likelihood
Gradient of log-likelihood
Sample the permutations from P(σ|G,Θ) and average the gradients

25. Sampling node correspondences
Metropolis sampling:
Start with a random permutation
Do local moves on the permutation
Accept the new permutation
If new permutation is better (gives higher likelihood)
If new is worse accept with probability proportional to the ratio of likelihoods 1
Swap node labels 1 and 4 4
Re-evaluate the likelihood 3 3 2 2 4 1
Can compute efficiently:
Only need to account for changes in 2 rows / columns 1 2 3 4 1 4 2 3 1 1

26. Solution 2: Calculating P(G|Θ,σ)
Calculating naively P(G|Θ,σ) takes O(N2)
Idea:
First calculate likelihood of empty graph, a graph with 0 edges
Correct the likelihood for edges that we observe in the graph
By exploiting the structure of Kronecker product we obtain closed form for likelihood of an empty graph

27. Solution 2: Calculating P(G|Θ,σ)
We approximate the likelihood:
The sum goes only over the edges
Evaluating P(G|Θ,σ) takes O(E) time
Real graphs are sparse, E << N2
Empty graph
No-edge likelihood
Edge likelihood
ADD ANIMATION

28. Experiments: synthetic data
Can gradient descent recover true parameters?
Optimization problem is not convex
How nice (without local minima) is optimization space?
Generate a graph from random parameters
Start at random point and use gradient descent
We recover true parameters 98% of the times

29. Convergence of properties
How does algorithm converge to true parameters with gradient descent iterations?
Log-likelihood
Avg abs error
Gradient descent iterations
Gradient descent iterations
1st eigenvalue
Diameter

30. Experiments: real networks
Experimental setup:
Given real graph
Stochastic gradient descent from random initial point
Obtain estimated parameters
Generate synthetic graphs
Compare properties of both graphs
We do not fit the properties themselves
We fit the likelihood and then compare the graph properties

31. AS graph (N=6500, E=26500) Autonomous systems (internet)
We search the space of ~1050,000 permutations
Fitting takes 20 minutes
AS graph is undirected and estimated parameter matrix is symmetric:
0.98
0.58
0.06

32. AS: comparing graph properties
Generate synthetic graph using estimated parameters
Compare the properties of two graphs
Degree distribution
Hop plot
diameter=4
log # of reachable pairs
log count
log degree
number of hops

33. AS: comparing graph properties
Spectral properties of graph adjacency matrices
Scree plot
Network value
log eigenvalue
log value
log rank
log rank

34. Epinions graph (N=76k, E=510k)
We search the space of ~101,000,000 permutations
Fitting takes 2 hours
The structure of the estimated parameter gives insight into the structure of the graph
0.99
0.54
0.49
0.13
Degree distribution
Hop plot
log count
log # of reachable pairs
log degree
number of hops

35. Epinions graph (N=76k, E=510k)
Scree plot
Network value
log eigenvalue
log rank
log rank

36. Scalability
Fitting scales linearly with the number of edges

37. Conclusion Kronecker Graph model has
provable properties
small number of parameters
We developed scalable algorithms for fitting Kronecker Graphs
We can efficiently search large space (~101,000,000) of permutations
Kronecker graphs fit well real networks using few parameters
We match graph properties without a priori deciding on which ones to fit

38. References
Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations, by Jure Leskovec, Jon Kleinberg, Christos Faloutsos, ACM KDD 2005
Graph Evolution: Densification and Shrinking Diameters, by Jure Leskovec, Jon Kleinberg and Christos Faloutsos, ACM TKDD 2007
Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication, by Jure Leskovec, Deepay Chakrabarti, Jon Kleinberg and Christos Faloutsos, PKDD 2005
Scalable Modeling of Real Graphs using Kronecker Multiplication, by Jure Leskovec and Christos Faloutsos, ICML 2007
Acknowledgements: Christos Faloutsos, Jon Kleinberg, Zoubin Gharamani, Pall Melsted, Alan Frieze, Larry Wasserman, Carlos Guestrin, Deepay Chakrabarti