EME: Information Theoretic views of emergence and self-organisation Continuing the search for useful definitions of emergence and self organisation

EME : 2 The plot so far … Researchers agree on just two things –We need a consistent definition of emergence –We don’t have one Statistical complexity and mutual information help us to explore definitions in information theoretic terms –So long as we can extract a Shannon-compliant representation of information in a system So, now we can give formal definitions of emergence and self-organisation –And to explore the relationships among them…

EME : 3 Entropy: recap and use Joint entropy Measure of information in joint systems Conditional entropy (mutual information) Measure of information in a system relative to that in another –Evolution increases mutual information between a system and its environment Entropy can be compared –between systems, or, for one system: over space, or over time Entropy measures to find how much emergent information is a direct consequence of low-level information If we can encode the information appropriately –But that does not help to define emergence

EME : 4 Phase transitions: recap and use System behaviour is most complex at phase transitions –Many emergent and self-organising phenomena are associated with complexity First order transitions generate latent heat, entropy, and complex behaviour Turbulence and related behaviours Second order transitions are not abrupt changes in entropy But important in identifying complex behaviour (Ising model) Phase transitions are related to emergence –but do not help define it

EME : 5 Recognising emergence and self-organisation Emergence is essentially about the appearance of patterns at larger/higher scales than components Self-organisation is essentially about the “spontaneous” appearance of structure over time To define emergence and self-organisation, we need to identify pattern or structure, and compare it across space or time –Complexity measures can be used in their definition So long as the measures distinguish complexity from randomness

EME : 6 Emergence and complexity Emergence and self-organisation can be defined using statistical complexity –Used by Crutchfield (1994), and subsequently by Shalizi (2001) –Based on the ability of a measurable system (e.g., an ε- machine) to reproduce the statistical information characteristics of an actual system Essentially, looking for a robust way to identify and measure structure, or pattern Crutchfield, The Calculi of Emergence, Physica D, 75, 1994 Shalizi, May 2001, Prokopenko et al, An information-theoretic primer…, Advances in Complex Systems, 2006

EME : 7 Argument underlying definitions Exact models of the structure inherent in systems are uncomputable in general, and uninformative Needs a UTM, or a simulated universe, and precise measurement… Much low-level detail is unnecessary to approximate higher-level behaviour As in thermodynamics Approximations improve when statistical analysis of repeated measurement and recalculation is used Best representation of average properties “Given the ubiquity of noise in nature, this is a small price to pay” Shalizi, May 2001,

EME : 8 Shalizi’s definition of emergence A derived process is emergent if it has a greater predictive efficiency than the process it derives from Predictive efficiency e relates excess entropy, E and statistical complexity, C μ : e = E / C μ –C μ is amount of memory of past stored in a process or system –E is mutual information between system’s past and future amount of apparent information about past stored in observed behaviour –E ≤ C μ, so 0 ≤ e < 1 e = the fraction of historical memory stored in the process which does “useful work” in telling us about the future –C μ = 0 if no complexity, so no predictive interest, so set e = 0 perfect predictive efficiency “unlikely” Shalizi, May 2001,

EME : 9 Intuitive interpretation of Shalizi’s definition A derived process is emergent if it has a greater predictive efficiency than the process it derives from For process X that emerges from process Y, e x > e y e = E / C μ So an emergent system has either lower statistical complexity or greater excess entropy Compared to system from which it emerges, an emergent system: –either has fewer irreducible complex or random components –or its past determines its future more completely

EME : 10 A informal check of Shalizi’s definition GoL glider vs CA with GoL rules –Glider’s past predicts its future absolutely –CA rule prediction depends on attractor space Slime mould: Dicty slug vs Dicty amoebae –Slug moves to favourable site and fruits –Amoeba may stay alive, become spore/cyst on its own, may become a slug pre-stalk or a slug pre-spore In these simple emergent behaviours predictive efficiency seems greater than the underlying system –… but what about all the others? –Exhaustive studies are not yet being done

EME : 11 Excess entropy calculation How do we calculate predictive efficiency, e = E / C μ ? Excess entropy (E) is mutual information between a system’s pasts and future –A system cannot have more mutual information than that in either the past or future states System’s pasts are represented by causal states –Equivalence class of input states that all have same conditional probability distribution of outputs Shalizi, May 2001,

EME : 12 Intuition for Causal States Causal states are produced by the modelling process –They are observations of the “state machine”, not the state machine itself Recreates a minimal model with equivalent statistical behaviour Uses a series of spatial or temporal measurements Causal states are not states of the actual system –Recall lecture 12: logistic process 47 deduced causal states Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

EME : 13 ε-machines and causal states From Crutchfield, ε-machines used to extract causal states from discrete time series Discrete measurements are approximate indicators of a hidden environment with finite accuracy, ε An ε-machine detects causal states by identifying pasts that predict the future –Based on computation theory and prediction of bit-strings Various algorithms for reconstructing an ε-machine Shalizi and Crutchfield give example calculations Crutchfield, The Calculi of Emergence, Physica D, 75, 1994 Shalizi, May 2001,

EME : 14 ε-machine: optimal model of complexity We cannot measure complexity directly –An ε-machine approximates the system’s information processing –An ε-machine is the smallest possible explanatory model Ockham’s razor – include only what is needed –Maximal accounting for structure Basic tenet of science – obtain prediction of nature Achieve appropriate balance between order and randomness Compromise between: –Smallest model with huge error ε and little prediction –Model with minimal error that differs minimally from system Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

EME : 15 Can we “calculate” emergence? It is hard to calculate causal states in general In Markovian behaviour (e.g. thermodynamics) any pre-state is a causal state –Standard entropy formulae in thermodynamics and statistical mechanics allow calculation of predictive efficiency –Not surprisingly, the predictive efficiency of macro-scale is considerably better than that of the micro-scale independent gas particles with almost no predictive power give rise to predictable emergent behaviour expressible in a small number of macro-variables So, by this measure, thermodynamics is emergent Shalizi, May 2001,

EME : 16 A note on calculations for thermodynamics Shalizi’s predictive efficiency calculation uses recognised facts and formulae of thermodynamics and statistical mechanics –Many assumptions about quantities, accuracy, etc. e.g. assumes macro-measurement error factor < –Finds sub-nano-second predictive efficiency of micro-scale is high, but rapidly reduces over time Most information in statistical mechanics is irrelevant to thermodynamic macro-state –But the cumulated assumptions and errors make the figures at best debatable

EME : 17 Emergent structures and ε-machines An ε-machine summarises the dynamics of a process The ε-machine could be divided in to sub-machines and transitions among them –At each time step, causal and previous state may or may not be in same sub-machine If successive states are in one sub-machine, this is an emergent process of the process approximated by the ε-machine –Because knowing sub-machine reduces statistical complexity Shalizi, May 2001,

EME : 18 Shalizi’s definition of self-organisation An increase in statistical complexity is a necessary condition for self-organisation A more realistic definition, relying only on statistical complexity, not excess entropy, causal states, etc. –Successive states in different sub-machines of an ε-machine –Optimal prediction requires more information There are more irreducible complex or random components For non-stationary processes, distribution of causal states changes over time –Statistical complexity is measured as a function of time, C μ (t) –A system that spontaneously moves from uniform to periodic behaviour exhibits an increase in C μ (t) Shalizi, May 2001,

EME : 19 Thermodynamics: a box of gas In the micro-state, all particles are independent, and all behaviours equally likely –What happens in one time or spatial unit has no later effect Statistical complexity remains constant, and low Thermodynamic micro-process is not self-organising Note that, at least in this case, the definitions allow emergence without self-organisation –But thermodynamics is perhaps an extreme case Shalizi, May 2001,

EME : 20 Summarising the definitions Emergence –Informally: behaviour observed at one scale is not apparent at other scales –Formally: processes have better predictive efficiency than those from which they emerge Lower statistical complexity or greater explanatory power of the past Self-organisation –Informally: structures that emerge without systematic external stimuli –Formally: processes with an increase in statistical complexity over time

EME : 21 Intuitions on flocking Individual birds are probably not independent, but follow simple local rules –For a collection of birds, statistical complexity > 0 but no where near 1 If a bird could see the whole flock it would see complex dynamics –For a flock of birds, statistical complexity is closer to 1 The flock is a self-organised collection of birds When we recognise self-organisation we label it as an emergent behaviour at a higher scale –“Flock” denotes an emergent pattern of behaviour Predictive efficiency of flock is better than that of a group of birds

EME : 22 Reconciling the definitions Natural systems can self-organise –Independent of observation Systems that self-organise are studied by “cognitively-limited observers” –Seeking descriptions that have good predictive ability –Patterns are at a more abstract level than self-organising system elements An abstract description with enhanced prediction becomes an emergent process of the original behaviour

EME : 23 So does self-organisation imply emergence? Shalizi states that he knows of no reason against self- organising non-emergent systems, but … Emergence may be a precondition of detectable self- organisation –In practice, when humans recognise self-organisation, they identify the abstract result at an emergent process At least some of what humans call “noise” may be unrecognisable complex self-organisation –We know that chaos has complex structure … Shalizi, May 2001,

EME : 24 Self-organisation and emergence Not self- organising Self- organising Not emergentEmergent Not interesting Thermodynamics Possibly very complex self-organisation EMER interest: Biology Interesting physics … Improved modeling and understanding

EME : 25 So where are we now? Statistical complexity gives us a nice definition of self-organisation –And an intuition for how self-organisation and emergence are related Statistical complexity gives us a possible definition of emergence –In reality, predictive efficiency is hard to estimate with any confidence The definition does clarify features of emergence The definitions make assumptions about measurement and discrete spatial or temporal series

EME : 26 A note on discrete measurement All work on information theoretic definitions of emergence and self-organisation assumes discrete temporal or spatial observation Shalizi discusses using causal states with continuous trajectories: –No current mathematics of continuous conditional probability –Continuous entropy exists but depends on co-ordinates Entropy changes if e.g., distance is measured in inches or metres –Reconstruction from data is hard, and seriously affected by measurement problems Not unlike problems of using PDEs for explanatory models of biological systems

EME : 27 Open questions Observers and intrinsic emergence: If emergence is a precondition of detectable self- organisation, then it must be possible to observe the system In intrinsic emergence, the observer is a sub-process of the system monitors environment through sensors to construct an imperfect behavioural model –Observer makes predictions of future which, because internal, can interfere with that future But, are intrinsic observation and self-organisation compatible? –does observer have to be outside a self-organising system?

EME : 28 Open questions How to determine agency in self-organisation: Organisation assumes that structure emerges over time Self-organisation assumes that there is no consistent external agency in the appearance of structure But, how can we distinguish self-organisation from organisation by external agency? –A complex system in a complex environment, where external inputs may be stochastic, but might also be very complex –Biological systems are usually in this category

EME : 29 And one last open question Informally, research has identified level, scope and resolution as relevant to emergence –Statistical complexity and mutual information assume that processes or systems are well-defined –Statistical analysis (e.g., for causal states or PDEs) also assumes that scope is known What happens if the chosen bounds exclude a key component of the emergence or self-organisation? What happens if slightly widening the bounds would reveal that emergence or self-organisation was an artifice of the scope? We have raised as many questions as we have solved