Physics and Machine Learning “All the tricks that physicists’ use eventually end up in machine learning”

Energy – Physics definitions Energy - A measure of being able to do work. There are many forms of energy, such as heat, mechanical, electrical, radiant, chemical, and nuclear energies. Energy is measured in such units as the joule (J), erg, kilowatt- hour (kW-hr), kilocalorie (kcal), foot-pound (ft-lb.), electron-volt (ev), and British thermal unit (BTU). –NASA.gov "It is important to realize that in physics today, we have no knowledge of what energy is. We do not have a picture that energy comes in little blobs of a definite amount." -Richard Feynman "Lectures on Physics"

Energy - Physics II Images: Magnetic Arrays and Spin

Energy - Machine Learning H=-½  iJ w iJ S i S J SiSi SJSJ W iJ Vs.

Energy - Machine Learning II Energy is the difference in weight between all nodes that agree and all nodes that disagree. The more weights, the greater energy. The “closer” the call, the lower |H|.

Energy Minima Retrieval States – attractors Mixture States – linear combinations of odd numbered attractors Spin Glass States – uncorrelated to attractors.

Ferromagnetics

Energy Metaphor Imagine the atomic magnets as movable objects with the freedom to flip, but you control their position. Each iteration of learning is like forcing all magnets to be closer together, as such the network energy is potential energy and the flipping of spins is the expression of kinetic energy.

Temperature – Physics I Extending the ferromagnetic example As temperature increases, the impact of other atomic magnets’ spins is decreased. At absolute zero, temperature has no impact. At the critical temperature(T c ), spin has no impact.

Temperature – Ferromagnetics S i = +1 w/ probability g(h i ); else -1 g(h) = 1/(1+exp(-2βh)) β = 1/(k B T) k B = Boltzman’s constant T = temperature F β (+/-h i )=1/(1+exp(-/+ 2βh i ) F β (+/-h i ) F β (+/-h i ) is a logistic function

Temperature – Machine Learning Logistic function Noise Used in the elimination of spurious local minima

Mean Field Theory – Physics The individual measurement and summation of each member of a magnetic array is too expensive Physicists look to average values as an inexpensive way to extract further truth from a complex combinatronics problem.

Mean Field Theory – Physics h i =  J w iJ S J +h ext =  J w iJ +h ext =tan(β ) = tanh(β  J w iJ +h ext ) =tanh(βJ(S)) Tc T 1

Mean Field Theory – stochastic model =tanh(β/N  Ju ζ u i ζ u J ) We allow an assumption, that is proportional to one of the stored patterns ζ v i =m ζ v i =½N(1+m) Tc T

Mean Field Theory =½N(1+m) There is a point at which noise overcomes the ability of a network to make an informed decision.

Conclusions All these metaphors that pull from physics are very tightly linked to energy. All metaphors concentrate on atomic events. (exception that proves the rule: mean field theory).

Extra Time? Extra Topics!

Entropy – Physics The inevitable progression toward chaos The motion of energy and matter away from an organized state.

Entropy – Machine Learning S = -   P  logP  For S = -   P  log 2 P  (the binary case) this is the average amount of additional information required to specify one of the staties.

Quantum Mechanics Fits within the context of our expectation for where to look for Physics crossover Atomic – discrete and binary Energy specific Last Class Lecture – use of Dyads.