1. Power law distribution
Theory and practice
Presentation arrangement
Y. Twitto
Presented
November 2015

2. Outline Introduction Identifying power law behavior
The mathematics of power laws
Emergence of power law distributions
Phase transitions
Self-organized criticality
29-Nov-2015

3. Source and credits
Before we start
29-Nov-2015

4. Paper: Power laws, Pareto distributions and Zipf’s law
Author: M. E. J. Newman
Journal: Contemporary physics 46.5 (2005):
Citations: 3153 papers
28 pages monograph
29-Nov-2015

5. Author: M. E. J. Newman British physicist
Professor of Physics at the University of Michigan
External faculty member of the Santa Fe Institute
Contributed to the fields of
complex networks 
complex systems
Awarded the 2014 Lagrange Prize
29-Nov-2015

6. Journal: Contemporary physics
Country: United Kingdom
Subject Area: Physics and Astronomy
29-Nov-2015

7. Introduction
Examples and definition
29-Nov-2015

8. Heights of humans Center value
Most adult human beings are about 175cm tall
There is some variation around this figure
But we never see people who are 10cm tall, or 500cm
29-Nov-2015

9. Heights of humans Max to min ratio
World’s tallest man: 272cm
World’s shortest man: 57cm
Tallest to shortest ratio: 4.8
This ratio is relatively low
29-Nov-2015

10. Variation Not all things we measure are peaked around a typical value
Some vary over many orders of magnitude
A classic example of this type of behavior is the sizes of towns and cities
29-Nov-2015

11. Sizes of cities Max to min ratio
2700 US cities
With more that 10,000 citizens
New York City: 8M humans
Duffield, Virginia: 52 humans
Max to min ratio: 150,000
This ratio is relatively high
Data from the 2000 US census
29-Nov-2015

12. Sizes of cities Histogram shape
The histogram is highly right-skewed
The bulk of the distribution occurs for fairly small sizes
There is a small number of cities with population much higher than the typical value
A long tail to the right of the histogram
29-Nov-2015

13. Sizes of cities Histogram shape
USA’s total population: 300M
Thus, you could at most have about 40 cities the size of New York City
29-Nov-2015

14. Logarithmic scale Scales are multiplicative Instead of additive
29-Nov-2015

15. Sizes of cities Logarithmic scale
Approximately a line
Let 𝒑 𝒙 𝑑𝑥 be the fraction of cities with population between 𝑥 and 𝑥+𝑑𝑥
Being a straight line in log-log scale means
𝐥𝐧 𝒑(𝒙) =−𝜶 𝐥𝐧 𝒙 +𝒄
Where 𝛼 and 𝑐 are constants
29-Nov-2015

16. Sizes of cities Logarithmic scale
Taking the exponential of both sides, we obtain:
𝒑 𝒙 = 𝑪 𝒙 −𝜶
where 𝐶= 𝑒 𝑐
Distribution of this form is a power law distribution
29-Nov-2015

17. Identifying power law behavior
How?
29-Nov-2015

18. Identifying power law behavior
Can be tricky
Standard strategy
make a simple histogram, and plot it on log scale to see if it looks straight
Can serve as a preliminary test
But is not good enough in general
29-Nov-2015

19. Simulation results Tail is noisy because of sampling errors
Each bin has only a few samples in it, if any
Million random real numbers
Drawn from a power-law distribution: 𝒑 𝒙 = 𝑪 𝒙 −𝟐.𝟓
Bins size: 0.1, i.e., [1,1.1); [1.1, 1.2); [1.2, 1.3); and so on…
So fluctuations are large
29-Nov-2015

20. Simulation results We cannot throw the tail
Many distributions follows a power law only in the tail
Million random real numbers
Drawn from a power-law distribution: 𝒑 𝒙 = 𝑪 𝒙 −𝟐.𝟓
Bins size: 0.1, i.e., [1,1.1); [1.1, 1.2); [1.2, 1.3); and so on…
29-Nov-2015

21. Logarithmic binning Vary the width of the bins
Normalize the sample counts
Divide by the width of the bin
We get count per unit interval
Bins sizes: 0.1, 0.2, 0.4, 0.8, …
29-Nov-2015

22. Logarithmic binning Bins in the tail get more samples
This reduces the statistical errors
The histogram is much clearer
The histogram extended for at least ten times than before
29-Nov-2015

23. Logarithmic binning Still, some noise in the tail… Denote
𝒙 𝒎𝒊𝒏 : the bottom of the lowest bin
𝒂: the ratio of the widths of successive bins
Then the 𝑘’th bin limits are:
𝒙 𝒌−𝟏 = 𝑥 𝑚𝑖𝑛 𝑎 𝑘−1
𝒙 𝒌 = 𝑥 𝑚𝑖𝑛 𝑎 𝑘
29-Nov-2015

24. Logarithmic binning
As long as 𝜶>𝟏, the number of samples per bin goes down
as 𝑘 increases
Thus, the bins in the tail will have more statistical noise
Most power-law distributions in nature have 2≤𝛼≤3
so noisy tails are the norm
29-Nov-2015

25. Examining the CDF CDF = cumulative distribution function
A superior method, in many ways
But, no longer a simple representation of the distribution of the data
Plot 𝑷 𝒙 instead of 𝒑(𝒙)
29-Nov-2015

26. Examining the CDF CDF also follows power law!
With exponent 𝜶−𝟏
If we plot 𝑃(𝑥) on logarithmic scales we get a straight line again
No need to bin the data to calculate and plot 𝑃(𝑥)
It is well defined for every single value of 𝑥
𝑷(𝒙) can be plotted as a perfectly normal function without binning
29-Nov-2015

27. Examining the CDF Closer to a straight line
Cumulative distributions with a power-law form are sometimes said to follow Zipf ’s law or a Pareto distribution
Since power-law cumulative distributions imply a power-law form for p(x), “Zipf’s law” and “Pareto distribution” are effectively synonymous with “power-law distribution”
29-Nov-2015

28. Calculating the exponent
In our simulation, we use 𝛼=2.5
But in practical situations the value of 𝛼 is unknown
we want to estimate 𝛼 from observed data
29-Nov-2015

29. Calculating the exponent Commonly used method
Most commonly used method: fit the slope of the line in one of the plots
Unfortunately, it is known to introduce systematic biases into the value of the exponent
Thus the author suggests not to rely on such methods
A least-squares fit of a straight line to our plot gives 𝜶 = 𝟐.𝟐𝟔 ± 0.02
clearly incompatible with the known value of 𝜶 = 𝟐.𝟓
29-Nov-2015

30. Calculating the exponent Better way
𝒙 𝟏 , 𝒙 𝟐 ,…, 𝒙 𝒏
The measured values of 𝑥
𝒙 𝒎𝒊𝒏
The minimum value of 𝑥
For which the power-law behavior holds
Applying this approximation in our case we obtain 𝜶=𝟐.𝟓𝟎𝟎±𝟎.𝟎𝟎𝟐
Approximation for 𝜶
The error of the approximation
29-Nov-2015

31. Identified power law behavior
Examples
29-Nov-2015

32. Word frequency The frequency with which words are used
examined in depth and confirmed by Zipf
Figure: the cumulative distribution of the number of times that words occur in a typical piece of English text
In this case the text of the novel Moby Dick by Herman Melville
Moby Dick by Herman Melville
29-Nov-2015

33. Citations of scientific papers
The numbers of citations received by scientific papers
Data are taken from the Science Citation Index
For papers published in 1981
Figure: the cumulative distribution of the number of citations received by a paper
between publication and June 1997
𝒙 𝒎𝒊𝒏
papers published in 1981
29-Nov-2015

34. Wealth of the richest people
The cumulative distribution of the total wealth of the richest people in the United States
As for 2003
Bill Gates
$46B
29-Nov-2015

35. 𝑋 min
Few real-world distributions follow a power law over their entire range
In particular not for smaller values of the variable being measured
The function 𝒑 𝒙 = 𝑪 𝒙 −𝜶 diverges as 𝒙→𝟎
For any positive value of α
In reality, the distribution must deviate from the power-law form below some minimum value 𝒙 𝒎𝒊𝒏
Thus one often hears it said that the distribution of such-and-such a quantity “has a power-law tail”
29-Nov-2015

36. 𝑋 min
Extracting a value for 𝜶 from real-world distributions can be tricky
It requires us to make a judgement, about the value 𝒙 𝒎𝒊𝒏 above which the distribution follows the power law
29-Nov-2015

37. Estimates for α The estimated exponents for several distributions
along with the values of 𝒙 𝒎𝒊𝒏 used in their calculations
29-Nov-2015

38. The mathematics of power laws
Normalization, moments, and largest value
29-Nov-2015

39. Normalization Or what is the value of C?
The constant C of the power law distribution is given by the normalization requirement of 𝑝(𝑥)
29-Nov-2015

40. Mean value Exists only if 𝛼>2
// 𝜶>𝟐
Mean value Exists only if 𝛼>2
Example of distributions with infinite mean: sizes of solar flares and wars
29-Nov-2015

41. Mean value What does it mean to say that a distribution has an infinite mean?
𝜶≤𝟐
Surely we can take the data for real solar flares and calculate their average
Each individual measurement is a finite number
There are a finite number of measurements
Only if we will have a truly infinite number of samples, we will see that the mean actually diverge
29-Nov-2015

42. Mean value What does it mean to say that a distribution has an infinite mean?
𝜶≤𝟐
If we were to repeat our finite experiment many times and calculate the mean for each repetition
Then the mean of those many means is itself also formally divergent
since it is simply equal to the mean we would calculate if all the repetitions were combined into one large experiment
29-Nov-2015

43. Mean value What does it mean to say that a distribution has an infinite mean?
𝜶≤𝟐
While the mean may take a relatively small value on any particular repetition of the experiment, it must occasionally take a huge value
Thus there must be very large fluctuations in the value of the mean
While we can quote a figure for the average of the samples we measure, that figure is not a reliable guide to the typical size of the samples in another instance of the same experiment
29-Nov-2015

44. Mean value
For 𝜶>𝟐 however, the mean is perfectly well defined
𝜶>𝟐
29-Nov-2015

45. The second moment Calculated similarly Diverges if 𝜶≤𝟑
Distributions in this range include almost all distributions appear in the table we saw previously
Such distributions have no meaningful variance
Or standard deviation
29-Nov-2015

46. The second moment
Well defined for 𝜶>𝟑
𝜶>𝟑
29-Nov-2015

47. The 𝑚’th moment
𝜶>𝟐
𝜶>𝒎−𝟏
𝜶>𝟑
29-Nov-2015

48. Largest value
Suppose we draw n measurements from a power-law distribution
What value is the largest of those measurements likely to take?
29-Nov-2015

49. Largest value 𝝅 𝒙 𝒏 𝒑(𝒙) 𝟏−𝑷 𝒙 𝒏−𝟏
The probability that the largest value falls in the interval 𝑥,𝑥+𝑑𝑥 𝒏
The number of options to choose the sample that will be the largest
𝒑(𝒙)
We want the chosen sample to fall in the interval 𝑥,𝑥+𝑑𝑥
𝟏−𝑷 𝒙 𝒏−𝟏
We want all the other samples to be smaller than the chosen sample
29-Nov-2015

50. Largest value 𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample
29-Nov-2015

51. Largest value 𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample
The Eulerian integral of the first kind
Also known as Legendre’s beta-function
Largest value
𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample
29-Nov-2015

52. Largest value Legendre’s beta-function 𝐵(𝑎,𝑏)
For large 𝑎 and fixed 𝑏: 𝑩 𝒂,𝒃 ~ 𝒂 −𝒃
In our case: 𝑩 𝒏, 𝜶−𝟐 𝜶−𝟏 ~ 𝒏 − 𝜶−𝟐 𝜶−𝟏
Thus: 𝒙 𝒎𝒂𝒙 ~ 𝒏 𝟏 𝜶−𝟏
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/2
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/10
𝒙 𝒎𝒂𝒙 grows larger as long as 𝜶>𝟏
29-Nov-2015

53. The mathematics of power laws Summary
𝒙 𝒎𝒂𝒙 ~ 𝒏 𝟏 𝜶−𝟏
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/2
𝛼= 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/10
𝜶>𝟏
𝜶>𝟏
29-Nov-2015
𝜶>𝟐
𝜶>𝟑

54. Emergence of power law distributions
Phase transitions & self-organized criticality
29-Nov-2015

55. Phase transitions
Topic that has received a huge amount of attention over the past few decades from the physics community
Some systems have only a single macroscopic parameter governing them
Sometimes, when this parameter grows larger the system diverges
The precise point at which divergence occurs is called the system critical point or the system phase transition
29-Nov-2015

56. Phase transitions The percolation transition
𝑝=1/2
We will explore one instructive example: the “percolation transition”
Lattice
Squares
Coloring probability 𝒑 (indep.)
Clusters
29-Nov-2015

57. Phase transitions The percolation transition
𝑝=1/2
What is the mean area 𝒔 of the cluster to which a randomly chosen square belongs?
Notice that the above is not the mean area of a cluster
29-Nov-2015

58. Phase transitions Small 𝑝
When 𝒑 is small, only a few squares are colored in
Most colored squares will be alone on the lattice
Or maybe grouped in twos or threes
So 𝒔 will be small
𝑝=1/2
29-Nov-2015

59. Phase transitions Large 𝑝
If 𝑝 is large, then most squares will be colored
Almost all the squares will be connected together
In one large cluster
In this situation we say that the system percolates
As we let the lattice size become larger, 𝒔 also becomes larger
𝑝=1/2
29-Nov-2015

60. Phase transitions Large 𝑝
Now the mean size of the cluster to which a random square belongs is limited only by the size of the lattice
As we let the lattice size become larger, 𝒔 also becomes larger
𝑝=1/2
29-Nov-2015

61. Phase transitions Two behaviors
So we have two distinctly different behaviors
One for small 𝑝 in which 𝒔 is small and doesn’t depend on the size of the system
One for large 𝑝 in which 𝒔 is much larger and increases with the size of the system
𝑝=1/2
29-Nov-2015

62. Phase transitions Two behaviors
What happens in between these two extremes?
As we increase 𝑝 from small values, the value of 𝒔 also increases
at some point we reach the start of the regime in which 𝒔 goes up with system size
This point is at 𝒑= 𝒑 𝒄𝒓𝒊𝒕𝒊𝒄𝒂𝒍 = 𝒑 𝒄 = 𝟎.𝟓𝟗𝟐𝟕𝟒𝟔𝟐…
Calculated from an average over 1000 simulations on a 1000×1000 square lattice. The dotted line marks the known position of the phase transition
29-Nov-2015

63. Phase transitions 𝑝=0.3 𝑝= 𝑝 𝑐 =0.5927 𝑝=0.9
Cumulative distribution of the sizes of clusters for a square lattice of size 40000×40000 at the critical site occupation probability 𝑝 𝑐 =
Phase transitions
Three examples of percolation systems
on 100×100 square lattices
29-Nov-2015

64. Self-organized criticality
The forest fire model
29-Nov-2015

65. Self-organized criticality
Some dynamical systems arrange themselves so that they always sit at the critical point
Regardless their initial state
A example of such a system is the forest fire model
29-Nov-2015

66. Self-organized criticality Forest fire model
Consider the percolation model as a primitive model of a forest
The lattice represents the landscape
A single tree can grow in each square
Occupied squares represent trees
Empty squares represent empty land with no trees
29-Nov-2015

67. Self-organized criticality Forest fire model
Trees appear instantaneously at random at some constant rate
hence the squares of the lattice fill up at random
Every once in a while a wildfire starts at a random square on the lattice
set off by a lightning strike perhaps
The fire burns the tree in that square, if there is one, along with every other tree in the cluster connected to it
29-Nov-2015

68. Self-organized criticality Forest fire model
After a fire, trees can grow up again in the squares vacated by burnt trees
So the process keeps going indefinitely
29-Nov-2015

69. Self-organized criticality Forest fire model
We start with a empty lattice
Trees start to appear
Initially the forest will be sparse
Lightning strikes will either hit empty squares or a small localized cluster
Because the clusters are small and localized as we are well below the percolation threshold
Thus fires will have essentially no effect on the forest
29-Nov-2015

70. Self-organized criticality Forest fire model
As time goes by, more and more trees will grow up
At some point, the system contains fairly large clusters
When any tree in a large cluster gets hit by the lightning the entire cluster will burn away
29-Nov-2015

71. Self-organized criticality Forest fire model
This gets rid of the spanning cluster so that the system does not percolate any more
But over time as more trees appear it will presumably reach percolation again
So the scenario will play out repeatedly
29-Nov-2015

72. Self-organized criticality Forest fire model
The end result is that the system oscillates right around the critical point
First going just above the percolation threshold as trees appear
And then being beaten back below it by fire
29-Nov-2015

73. Self-organized criticality Forest fire model
In the limit of large system size these fluctuations become small compared to the size of the system as a whole
And to an excellent approximation the system just sits at the threshold indefinitely
Thus, if we wait long enough, we expect the forest fire model to self-organize to a state in which it has a power- law distribution of the sizes of clusters, or of the sizes of fires
29-Nov-2015

74. Self-organized criticality Forest fire model
Cumulative distribution of the sizes of “fires” in a simulation of the forest fire model
For a square lattice of size 5000×5000
As we can see, it follows a power law closely
𝜶=𝟏.𝟏𝟗±𝟎.𝟎𝟏
cutoff
29-Nov-2015

75. Self-organized criticality Forest fire model
The exponent of the distribution is quite small in this case
The best current estimates give a value of 𝜶=𝟏.𝟏𝟗±𝟎.𝟎𝟏
Thus, the distribution has an infinite mean in the limit of large system size
29-Nov-2015

76. Self-organized criticality Forest fire model
For all real systems however the mean is finite
The distribution is cut off in the large-size tail
Because fires cannot have a size any greater than that of the lattice as a whole
This makes the mean well- behaved
cutoff
29-Nov-2015

77. Self-organized criticality
There has been much excitement about self-organized criticality
As a possible generic mechanism for explaining where power-law distributions come from
Per Bak, one of the originators of the idea, wrote an entire book about it
29-Nov-2015

78. Self-organized criticality
Self-organized critical models have been put forward not only for forest fires, but for earthquakes, solar flares, biological evolution, avalanches, and many other phenomena
29-Nov-2015

79. Summary
29-Nov-2015

80. Summary Introduction Normalization Sizes of cities Mean and moments
Logarithmic scale
Logarithmic binning
Examining the CDF
Calculating the exponent
Examples: words, citation, wealth
Normalization
Mean and moments
Largest value
Phase transition
Self-organized criticality
29-Nov-2015

81. Thanks! Introduction Normalization Sizes of cities Mean and moments
Logarithmic scale
Logarithmic binning
Examining the CDF
Calculating the exponent
Examples: words, citation, wealth
Normalization
Mean and moments
Largest value
Phase transition
Self-organized criticality
29-Nov-2015