Power law distribution Theory and practice Presentation arrangement Y. Twitto Presented November 2015

Outline Introduction Identifying power law behavior The mathematics of power laws Emergence of power law distributions Phase transitions Self-organized criticality 29-Nov-2015

Source and credits Before we start 29-Nov-2015

Paper: Power laws, Pareto distributions and Zipf’s law Author: M. E. J. Newman Journal: Contemporary physics 46.5 (2005): 323-351 Citations: 3153 papers 28 pages monograph 29-Nov-2015

Author: M. E. J. Newman British physicist Professor of Physics at the University of Michigan External faculty member of the Santa Fe Institute Contributed to the fields of complex networks  complex systems Awarded the 2014 Lagrange Prize 29-Nov-2015

Journal: Contemporary physics Country: United Kingdom Subject Area: Physics and Astronomy 29-Nov-2015

Introduction Examples and definition 29-Nov-2015

Heights of humans Center value Most adult human beings are about 175cm tall There is some variation around this figure But we never see people who are 10cm tall, or 500cm 29-Nov-2015

Heights of humans Max to min ratio World’s tallest man: 272cm World’s shortest man: 57cm Tallest to shortest ratio: 4.8 This ratio is relatively low 29-Nov-2015

Variation Not all things we measure are peaked around a typical value Some vary over many orders of magnitude A classic example of this type of behavior is the sizes of towns and cities 29-Nov-2015

Sizes of cities Max to min ratio 2700 US cities With more that 10,000 citizens New York City: 8M humans Duffield, Virginia: 52 humans Max to min ratio: 150,000 This ratio is relatively high Data from the 2000 US census 29-Nov-2015

Sizes of cities Histogram shape The histogram is highly right-skewed The bulk of the distribution occurs for fairly small sizes There is a small number of cities with population much higher than the typical value A long tail to the right of the histogram 29-Nov-2015

Sizes of cities Histogram shape USA’s total population: 300M Thus, you could at most have about 40 cities the size of New York City 29-Nov-2015

Logarithmic scale Scales are multiplicative Instead of additive 29-Nov-2015

Sizes of cities Logarithmic scale Approximately a line Let 𝒑 𝒙 𝑑𝑥 be the fraction of cities with population between 𝑥 and 𝑥+𝑑𝑥 Being a straight line in log-log scale means 𝐥𝐧 𝒑(𝒙) =−𝜶 𝐥𝐧 𝒙 +𝒄 Where 𝛼 and 𝑐 are constants 29-Nov-2015

Sizes of cities Logarithmic scale Taking the exponential of both sides, we obtain: 𝒑 𝒙 = 𝑪 𝒙 −𝜶 where 𝐶= 𝑒 𝑐 Distribution of this form is a power law distribution 29-Nov-2015

Identifying power law behavior How? 29-Nov-2015

Identifying power law behavior Can be tricky Standard strategy make a simple histogram, and plot it on log scale to see if it looks straight Can serve as a preliminary test But is not good enough in general 29-Nov-2015

Simulation results Tail is noisy because of sampling errors Each bin has only a few samples in it, if any Million random real numbers Drawn from a power-law distribution: 𝒑 𝒙 = 𝑪 𝒙 −𝟐.𝟓 Bins size: 0.1, i.e., [1,1.1); [1.1, 1.2); [1.2, 1.3); and so on… So fluctuations are large 29-Nov-2015

Simulation results We cannot throw the tail Many distributions follows a power law only in the tail Million random real numbers Drawn from a power-law distribution: 𝒑 𝒙 = 𝑪 𝒙 −𝟐.𝟓 Bins size: 0.1, i.e., [1,1.1); [1.1, 1.2); [1.2, 1.3); and so on… 29-Nov-2015

Logarithmic binning Vary the width of the bins Normalize the sample counts Divide by the width of the bin We get count per unit interval Bins sizes: 0.1, 0.2, 0.4, 0.8, … 29-Nov-2015

Logarithmic binning Bins in the tail get more samples This reduces the statistical errors The histogram is much clearer The histogram extended for at least ten times than before 29-Nov-2015

Logarithmic binning Still, some noise in the tail… Denote 𝒙 𝒎𝒊𝒏 : the bottom of the lowest bin 𝒂: the ratio of the widths of successive bins Then the 𝑘’th bin limits are: 𝒙 𝒌−𝟏 = 𝑥 𝑚𝑖𝑛 𝑎 𝑘−1 𝒙 𝒌 = 𝑥 𝑚𝑖𝑛 𝑎 𝑘 29-Nov-2015

Logarithmic binning As long as 𝜶>𝟏, the number of samples per bin goes down as 𝑘 increases Thus, the bins in the tail will have more statistical noise Most power-law distributions in nature have 2≤𝛼≤3 so noisy tails are the norm 29-Nov-2015

Examining the CDF CDF = cumulative distribution function A superior method, in many ways But, no longer a simple representation of the distribution of the data Plot 𝑷 𝒙 instead of 𝒑(𝒙) 29-Nov-2015

Examining the CDF CDF also follows power law! With exponent 𝜶−𝟏 If we plot 𝑃(𝑥) on logarithmic scales we get a straight line again No need to bin the data to calculate and plot 𝑃(𝑥) It is well defined for every single value of 𝑥 𝑷(𝒙) can be plotted as a perfectly normal function without binning 29-Nov-2015

Examining the CDF Closer to a straight line Cumulative distributions with a power-law form are sometimes said to follow Zipf ’s law or a Pareto distribution Since power-law cumulative distributions imply a power-law form for p(x), “Zipf’s law” and “Pareto distribution” are effectively synonymous with “power-law distribution” 29-Nov-2015

Calculating the exponent In our simulation, we use 𝛼=2.5 But in practical situations the value of 𝛼 is unknown we want to estimate 𝛼 from observed data 29-Nov-2015

Calculating the exponent Commonly used method Most commonly used method: fit the slope of the line in one of the plots Unfortunately, it is known to introduce systematic biases into the value of the exponent Thus the author suggests not to rely on such methods A least-squares fit of a straight line to our plot gives 𝜶 = 𝟐.𝟐𝟔 ± 0.02 clearly incompatible with the known value of 𝜶 = 𝟐.𝟓 29-Nov-2015

Calculating the exponent Better way 𝒙 𝟏 , 𝒙 𝟐 ,…, 𝒙 𝒏 The measured values of 𝑥 𝒙 𝒎𝒊𝒏 The minimum value of 𝑥 For which the power-law behavior holds Applying this approximation in our case we obtain 𝜶=𝟐.𝟓𝟎𝟎±𝟎.𝟎𝟎𝟐 Approximation for 𝜶 The error of the approximation 29-Nov-2015

Identified power law behavior Examples 29-Nov-2015

Word frequency The frequency with which words are used examined in depth and confirmed by Zipf Figure: the cumulative distribution of the number of times that words occur in a typical piece of English text In this case the text of the novel Moby Dick by Herman Melville Moby Dick by Herman Melville 29-Nov-2015

Citations of scientific papers The numbers of citations received by scientific papers Data are taken from the Science Citation Index For papers published in 1981 Figure: the cumulative distribution of the number of citations received by a paper between publication and June 1997 𝒙 𝒎𝒊𝒏 papers published in 1981 29-Nov-2015

Wealth of the richest people The cumulative distribution of the total wealth of the richest people in the United States As for 2003 Bill Gates $46B 29-Nov-2015

𝑋 min Few real-world distributions follow a power law over their entire range In particular not for smaller values of the variable being measured The function 𝒑 𝒙 = 𝑪 𝒙 −𝜶 diverges as 𝒙→𝟎 For any positive value of α In reality, the distribution must deviate from the power-law form below some minimum value 𝒙 𝒎𝒊𝒏 Thus one often hears it said that the distribution of such-and-such a quantity “has a power-law tail” 29-Nov-2015

𝑋 min Extracting a value for 𝜶 from real-world distributions can be tricky It requires us to make a judgement, about the value 𝒙 𝒎𝒊𝒏 above which the distribution follows the power law 29-Nov-2015

Estimates for α The estimated exponents for several distributions along with the values of 𝒙 𝒎𝒊𝒏 used in their calculations 29-Nov-2015

The mathematics of power laws Normalization, moments, and largest value 29-Nov-2015

Normalization Or what is the value of C? The constant C of the power law distribution is given by the normalization requirement of 𝑝(𝑥) 29-Nov-2015

Mean value Exists only if 𝛼>2 // 𝜶>𝟐 Mean value Exists only if 𝛼>2 Example of distributions with infinite mean: sizes of solar flares and wars 29-Nov-2015

Mean value What does it mean to say that a distribution has an infinite mean? 𝜶≤𝟐 Surely we can take the data for real solar flares and calculate their average Each individual measurement is a finite number There are a finite number of measurements Only if we will have a truly infinite number of samples, we will see that the mean actually diverge 29-Nov-2015

Mean value What does it mean to say that a distribution has an infinite mean? 𝜶≤𝟐 If we were to repeat our finite experiment many times and calculate the mean for each repetition Then the mean of those many means is itself also formally divergent since it is simply equal to the mean we would calculate if all the repetitions were combined into one large experiment 29-Nov-2015

Mean value What does it mean to say that a distribution has an infinite mean? 𝜶≤𝟐 While the mean may take a relatively small value on any particular repetition of the experiment, it must occasionally take a huge value Thus there must be very large fluctuations in the value of the mean While we can quote a figure for the average of the samples we measure, that figure is not a reliable guide to the typical size of the samples in another instance of the same experiment 29-Nov-2015

Mean value For 𝜶>𝟐 however, the mean is perfectly well defined 𝜶>𝟐 29-Nov-2015

The second moment Calculated similarly Diverges if 𝜶≤𝟑 Distributions in this range include almost all distributions appear in the table we saw previously Such distributions have no meaningful variance Or standard deviation 29-Nov-2015

The second moment Well defined for 𝜶>𝟑 𝜶>𝟑 29-Nov-2015

The 𝑚’th moment 𝜶>𝟐 𝜶>𝒎−𝟏 𝜶>𝟑 29-Nov-2015

Largest value Suppose we draw n measurements from a power-law distribution What value is the largest of those measurements likely to take? 29-Nov-2015

Largest value 𝝅 𝒙 𝒏 𝒑(𝒙) 𝟏−𝑷 𝒙 𝒏−𝟏 The probability that the largest value falls in the interval 𝑥,𝑥+𝑑𝑥 𝒏 The number of options to choose the sample that will be the largest 𝒑(𝒙) We want the chosen sample to fall in the interval 𝑥,𝑥+𝑑𝑥 𝟏−𝑷 𝒙 𝒏−𝟏 We want all the other samples to be smaller than the chosen sample 29-Nov-2015

Largest value 𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample 29-Nov-2015

Largest value 𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample The Eulerian integral of the first kind Also known as Legendre’s beta-function Largest value 𝒙 𝒎𝒂𝒙 -- the mean value of the largest sample 29-Nov-2015

Largest value Legendre’s beta-function 𝐵(𝑎,𝑏) For large 𝑎 and fixed 𝑏: 𝑩 𝒂,𝒃 ~ 𝒂 −𝒃 In our case: 𝑩 𝒏, 𝜶−𝟐 𝜶−𝟏 ~ 𝒏 − 𝜶−𝟐 𝜶−𝟏 Thus: 𝒙 𝒎𝒂𝒙 ~ 𝒏 𝟏 𝜶−𝟏 𝛼=2 𝑥 𝑚𝑎𝑥 ~ 𝑛 𝛼=3 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/2 𝛼=11 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/10 𝒙 𝒎𝒂𝒙 grows larger as long as 𝜶>𝟏 29-Nov-2015

The mathematics of power laws Summary 𝒙 𝒎𝒂𝒙 ~ 𝒏 𝟏 𝜶−𝟏 𝛼=2 𝑥 𝑚𝑎𝑥 ~ 𝑛 𝛼=3 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/2 𝛼=11 𝑥 𝑚𝑎𝑥 ~ 𝑛 1/10 𝜶>𝟏 𝜶>𝟏 29-Nov-2015 𝜶>𝟐 𝜶>𝟑

Emergence of power law distributions Phase transitions & self-organized criticality 29-Nov-2015

Phase transitions Topic that has received a huge amount of attention over the past few decades from the physics community Some systems have only a single macroscopic parameter governing them Sometimes, when this parameter grows larger the system diverges The precise point at which divergence occurs is called the system critical point or the system phase transition 29-Nov-2015

Phase transitions The percolation transition 𝑝=1/2 We will explore one instructive example: the “percolation transition” Lattice Squares Coloring probability 𝒑 (indep.) Clusters 29-Nov-2015

Phase transitions The percolation transition 𝑝=1/2 What is the mean area 𝒔 of the cluster to which a randomly chosen square belongs? Notice that the above is not the mean area of a cluster 29-Nov-2015

Phase transitions Small 𝑝 When 𝒑 is small, only a few squares are colored in Most colored squares will be alone on the lattice Or maybe grouped in twos or threes So 𝒔 will be small 𝑝=1/2 29-Nov-2015

Phase transitions Large 𝑝 If 𝑝 is large, then most squares will be colored Almost all the squares will be connected together In one large cluster In this situation we say that the system percolates As we let the lattice size become larger, 𝒔 also becomes larger 𝑝=1/2 29-Nov-2015

Phase transitions Large 𝑝 Now the mean size of the cluster to which a random square belongs is limited only by the size of the lattice As we let the lattice size become larger, 𝒔 also becomes larger 𝑝=1/2 29-Nov-2015

Phase transitions Two behaviors So we have two distinctly different behaviors One for small 𝑝 in which 𝒔 is small and doesn’t depend on the size of the system One for large 𝑝 in which 𝒔 is much larger and increases with the size of the system 𝑝=1/2 29-Nov-2015

Phase transitions Two behaviors What happens in between these two extremes? As we increase 𝑝 from small values, the value of 𝒔 also increases at some point we reach the start of the regime in which 𝒔 goes up with system size This point is at 𝒑= 𝒑 𝒄𝒓𝒊𝒕𝒊𝒄𝒂𝒍 = 𝒑 𝒄 = 𝟎.𝟓𝟗𝟐𝟕𝟒𝟔𝟐… Calculated from an average over 1000 simulations on a 1000×1000 square lattice. The dotted line marks the known position of the phase transition 29-Nov-2015

Phase transitions 𝑝=0.3 𝑝= 𝑝 𝑐 =0.5927 𝑝=0.9 Cumulative distribution of the sizes of clusters for a square lattice of size 40000×40000 at the critical site occupation probability 𝑝 𝑐 =0.592746 Phase transitions Three examples of percolation systems on 100×100 square lattices 29-Nov-2015

Self-organized criticality The forest fire model 29-Nov-2015

Self-organized criticality Some dynamical systems arrange themselves so that they always sit at the critical point Regardless their initial state A example of such a system is the forest fire model 29-Nov-2015

Self-organized criticality Forest fire model Consider the percolation model as a primitive model of a forest The lattice represents the landscape A single tree can grow in each square Occupied squares represent trees Empty squares represent empty land with no trees 29-Nov-2015

Self-organized criticality Forest fire model Trees appear instantaneously at random at some constant rate hence the squares of the lattice fill up at random Every once in a while a wildfire starts at a random square on the lattice set off by a lightning strike perhaps The fire burns the tree in that square, if there is one, along with every other tree in the cluster connected to it 29-Nov-2015

Self-organized criticality Forest fire model After a fire, trees can grow up again in the squares vacated by burnt trees So the process keeps going indefinitely 29-Nov-2015

Self-organized criticality Forest fire model We start with a empty lattice Trees start to appear Initially the forest will be sparse Lightning strikes will either hit empty squares or a small localized cluster Because the clusters are small and localized as we are well below the percolation threshold Thus fires will have essentially no effect on the forest 29-Nov-2015

Self-organized criticality Forest fire model As time goes by, more and more trees will grow up At some point, the system contains fairly large clusters When any tree in a large cluster gets hit by the lightning the entire cluster will burn away 29-Nov-2015

Self-organized criticality Forest fire model This gets rid of the spanning cluster so that the system does not percolate any more But over time as more trees appear it will presumably reach percolation again So the scenario will play out repeatedly 29-Nov-2015

Self-organized criticality Forest fire model The end result is that the system oscillates right around the critical point First going just above the percolation threshold as trees appear And then being beaten back below it by fire 29-Nov-2015

Self-organized criticality Forest fire model In the limit of large system size these fluctuations become small compared to the size of the system as a whole And to an excellent approximation the system just sits at the threshold indefinitely Thus, if we wait long enough, we expect the forest fire model to self-organize to a state in which it has a power- law distribution of the sizes of clusters, or of the sizes of fires 29-Nov-2015

Self-organized criticality Forest fire model Cumulative distribution of the sizes of “fires” in a simulation of the forest fire model For a square lattice of size 5000×5000 As we can see, it follows a power law closely 𝜶=𝟏.𝟏𝟗±𝟎.𝟎𝟏 cutoff 29-Nov-2015

Self-organized criticality Forest fire model The exponent of the distribution is quite small in this case The best current estimates give a value of 𝜶=𝟏.𝟏𝟗±𝟎.𝟎𝟏 Thus, the distribution has an infinite mean in the limit of large system size 29-Nov-2015

Self-organized criticality Forest fire model For all real systems however the mean is finite The distribution is cut off in the large-size tail Because fires cannot have a size any greater than that of the lattice as a whole This makes the mean well- behaved cutoff 29-Nov-2015

Self-organized criticality There has been much excitement about self-organized criticality As a possible generic mechanism for explaining where power-law distributions come from Per Bak, one of the originators of the idea, wrote an entire book about it 29-Nov-2015

Self-organized criticality Self-organized critical models have been put forward not only for forest fires, but for earthquakes, solar flares, biological evolution, avalanches, and many other phenomena 29-Nov-2015

Summary 29-Nov-2015

Summary Introduction Normalization Sizes of cities Mean and moments Logarithmic scale Logarithmic binning Examining the CDF Calculating the exponent Examples: words, citation, wealth Normalization Mean and moments Largest value Phase transition Self-organized criticality 29-Nov-2015

Thanks! Introduction Normalization Sizes of cities Mean and moments Logarithmic scale Logarithmic binning Examining the CDF Calculating the exponent Examples: words, citation, wealth Normalization Mean and moments Largest value Phase transition Self-organized criticality 29-Nov-2015