1. Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 14 Mean-Field Game
Zhu Han, Dusit Niyato, Walid Saad, and Tamer Basar
Thanks for Dr. Chungang Yang slides

2. Overview of Lecture Notes
Introduction to Game Theory: Lecture 1, book 1
Non-cooperative Games: Lecture 1, Chapter 3, book 1
Bayesian Games: Lecture 2, Chapter 4, book 1
Differential Games: Lecture 3, Chapter 5, book 1
Evolutionary Games: Lecture 4, Chapter 6, book 1
Cooperative Games: Lecture 5, Chapter 7, book 1
Auction Theory: Lecture 6, Chapter 8, book 1
Matching Game: Lecture 7, Chapter 2, book 2
Contract Theory, Lecture 8, Chapter 3, book 2
Learning in Game, Lecture 9, Chapter 6, book 2
Stochastic Game, Lecture 10, Chapter 4, book 2
Game with Bounded Rationality, Lecture 11, Chapter 5, book 2
Equilibrium Programming with Equilibrium Constraint, Lecture 12, Chapter 7, book 2
Zero Determinant Strategy, Lecture 13, Chapter 8, book 2
Mean Field Game, Lecture 14, UCLA course
Network Economy, Lecture 15, Dr. Jianwei Huang,

3. Outline Basics of MFG Applications of MFG
Conventional MFG in Ultra-Dense D2D Communications
Ultra-Dense Small Cell Networks: MFG with Interference Dominator
Mean Field Game-Theoretic Learning
Conclusion
What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need?
What is MFG?
Applications of MFG in networks, and in particular we

4. Mean Field Game, MFG
Basic Idea: While classical game theory models the interaction of a single player with all the other players of the system, a mean field game models the individual's interaction with the effect of the collective behavior of the players.
Think globally while behave locally
Think individually while behave locally
What is MFG? Why MFG?
Basically, MFG can be considered a special form of differential games with a large number of players.
Here, we know that conventional game models the interaction of one rational player with all the other players, however, the MFG models the any individual’s selfish behavior with respect to the collective behavior of all the other players. Here, the collective behavior is named as a mass.

5. Mean Field Game, MFG
Understand rational behaviors, and optimize their position in space and time dynamics.
The mean field theory is exploited to help decouple a complex large-scale optimization problem into a family of localized optimization problems.
Thus, each player can implement its policy by using only its local information.

6. Mean Field Game, MFG
Mean Field Game describes equilibrium configurations and dynamic behaviors of huge number of homogeneous agents, which is a class of differential games in which each agent is infinitesimal and interacts with a huge population of other agents.
Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, x) is the value function of a typical small player while m = m(t, x) denotes the mean field value at time t and at position x of the population.
Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system;
Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

7. Mean Field Game, MFG
The value function u satisfies a Hamilton-Jacobi-Bellman (HJB) equation—in which m enters as a parameter and describes the influence of the population on the cost of each agent, while the density m evolves in time according to a Fokker-Planck-Kolmogorov (FPK) equation in which u enters as a drift.
More precisely the pair (u,m) is a solution of the MFG system.
Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system;
Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

8. Mean Field Game, MFG
HJB equation governs the computation of the optimal path of control of the player, while FPK equation governs the evolution of the mean field function of players.
Here, the HJB and FPK equations are termed as the backward and forward functions, respectively.
Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system;
Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.
Combination of them in time creates some unusual phenomena in the time variable .

10. Outline Basics of MFG Applications of MFG
Conventional MFG in Ultra-Dense D2D Communications
Ultra-Dense Small Cell Networks: MFG with Interference Dominator
Mean Field Game-Theoretic Learning
Conclusion
What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need?
What is MFG?
Applications of MFG in networks, and in particular we

11. Challenges in 5G Ultra-Dense Networks
IoE: Internet of Everything
Mobile Internet
5G Ultra-Dense Small Cells
Social-Aware D2D Networks
Internet of Things
M2M, V2V Networks
5G Characteristics: Ultra-Dense, Scalable, Self-Organizing; Energy efficiency, spectral efficiency, cost efficiency
In 5G era, everything will be connected. It is called IoE for short. On the other hand, for better QoE of users, ultra-dense small cells and D2D networks,
IOT creating opportunities for more direct integration between the physical world and computer-based systems, and resulting in improved efficiency, accuracy and economic benefit.
Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications (M2M) and covers a variety of protocols, domains, and applications.[10] The interconnection of these embedded devices (including smart objects), is expected to usher in automation in nearly all fields, while also enabling advanced applications like a Smart Grid,[11] and expanding to the areas such as Smart city.

12. Challenges in 5G Ultra-Dense Networks
Technical Challenges
Coupled Systems & Interference -Limited Resources
Computationally Intractable & Analysis –Number Infinity
Low Convergence Rate & Robustness -Dynamics
Perfect Information & Backhauling Overhead -Others
Requirements
Self-Organization & Distributed Control
Space-time Dynamics

13. Game Theory in 5G Era
Extensive applications to characterize the rational behaviors, model the strategic interactions, and design distributed algorithms.
Face great challenges, e.g., the number of the nodes, e.g., access points and small cell base stations, is huge, even goes to infinity, leading to the well-known curse of dimensionality.
Hard to analyze the game due to the involved dynamics, complexity, interaction, signaling overhead, and so on.

14. Applications of MFG in Wireless Networks
Green Communications and Cognitive Radio Networks
H. Tembine, R. Tempone, and R. Vilanova, "Mean field games for cognitive radio networks", American Control Conference (ACC), Pages: , 2012.
F. Meriaux and S. Lasaulce, "Mean-field games and green power control", th International Conference on Network Games, Control and Optimization (NetGCooP), Pages: 1-5, 2011.
Hyper-Dense Heterogeneous Small Cell Networks
Prabodini Semasinghe and Ekram Hossain, Downlink Power Control in Self-Organizing DenseSmall Cells Underlaying Macrocells: A Mean Field Game, IEEE Transactions on Mobile Computing.
Ali Y. Al-Zahrani, F. Richard Yu, and Minyi Huang, A Joint Cross-Layer and Co-Layer Interference Management Scheme in Hyper-Dense Heterogeneous Networks Using Mean-Field Game Theory, IEEE Transactions on Vehicular Technology, 2015.
Cloud and Smart V2V Networks
A. F. Hanif, H. Tembine, M. Assaad, and D. Zeghlache, "Mean-field games for resource sharing in cloud-based networks", IEEE/ACMTransactions on Networking, 2015.
R. Couillet, S.M. Perlaza, H. Tembine, and M. Debbah, "Electrical Vehicles in the Smart Grid-A Mean Field Game Analysis", IEEE Journal on Selected Areas in Communications, Publication Year: 2012 , Page(s):

15. Distributed Interference and Energy Aware Power Control for Ultra-Dense D2D Networks: A Mean Field Game

16. Ultra-Dense D2D Networks
Device-to-device (D2D) provides significant performance enhancement in terms of spectrum and energy efficiency by proximity and frequency reuse. However, such performance enhancement is largely limited by mutual interference and energy availability, in particular, in ultra-dense D2D networks.
Powered by battery. Thus, an intelligent power control remaining energy of the battery.

17. Ultra-Dense D2D Networks
Interference dynamics and available energy, and then formulate a MFG theoretic framework with the interference mean-field approximation..
Investigate effects of both the interference dynamics of the generic device introduced to others, and all others' interference dynamics introduced into the generic device on power control.

18. D2D Differential Power Control Game
Remaining Energy
Interference Dynamics
Interference dynamics of the generic device introduced to others

19. D2D Differential Power Control Game
All others' interference dynamics introduced into the generic device
Instantaneous Cost function at time t, which is quadratic function of power consumption

20. Problem Formulation
Optimal control problem followed by Bellman's optimality principle.
Control Policy
Instantaneous Cost
Remaining Energy
State Space
Interference Dynamics

21. Solution
Hamilton-Jacobi-Bellman (HJB) : a partial differential equation…central to optimal control theory. Solution of HJB is the value function, which gives the minimum cost for a given dynamical system with an associated cost function.
Value function describes the minimum total cost from time t to time t+dt using specific policy.
Value function at time t+dt.
Cost function is determined by state and power at time t

22. Solution
Hamilton-Jacobi-Bellman (HJB) :

23. Mean-Field Game
Obtaining equilibrium for a system with N players involves solving N simultaneous partial differential equations.
State Dynamics and Number of Players
Mean field
Fokker-Planck-Kolmogorov (FPK) equation
Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players
involves solving $N$ simultaneous partial differential equations.

24. Mean-Field Game Mean field & FPK Equation
Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players
involves solving $N$ simultaneous partial differential equations.

25. MFG-Based Analysis
HJB Backward
FPK Forward
Hamiltonian
Results
Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, s) is the value function of a typical small player while m = m(t, s) denotes the mean field value at time t and at state s of the population.

26. MFG-Based Algorithm
Finite difference Technique in discretized way of space
Backward
Forward

27. Simulation Results
Extended Battery Life and Improved Energy Efficiency

28. Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator

29. Ultra-Dense Small Cell Networks
Extensive deployment of various small cells underlaying cellular networks reuses of the limited frequency resource to improve spectrum efficiency, thus interference.
Number tends to infinity, centralized interference management encounters signaling overhead, scalability, and flexibility, mainly due to random deployments and limited backhaul.
Therefore, decentralized interference management coupled with self-organizing properties should be studied, and that is why game theory has found extensive applications recently.

30. Ultra-Dense Small Cell Networks
Here, we choose SeNB1 as the generic player, and it will receive the both inter-tier interference and intra-tier interference. Here, we know the downlink power of the MeNB is stronger than that of the SeNBs, thus resulting MeNB as the interference dominator of the SeNB1 and other SeNBs as the minor interferers.

31. Mean Field Game with Interference Dominator
We jointly take the global and individual perspective to react to the interference when the number of small cell players tends to infinity.
To analyze and characterize the equilibrium behaviors, we formulate it as a mean field game, which simplify the analysis and facilitate distributed interference mitigation.
To think globally while behave locally, in practice an interference dominator always exists in addition to the mean field term, which would significantly affects the decision-making for the representative agent.
Therefore, we turn to mean field games between a dominating player and a group of representative agents, each of which acts similarly and also interacts with each other through a mean field term being substantially influenced by the interference dominating player.

32. Interference Mitigation Stochastic Game
With tow kinds of players: Interference dominator and minor interferers
Interference perceived by the interference dominator
Interference perceived by the minor generic interferer
Interference mean field:

33. Interference Mitigation Stochastic Game

34. Interference Mitigation Stochastic Game

35. MFG with Interference Dominator
We derive the HJB equation
We derive the FPK equation

36. Solution
Mean Field Update

37. Solution
Introduce Lagrangian Parameter to indirectly solve the HJB

38. Solution
Power Update

39. Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective

40. Strategic Mean Field Learning
What is distributed strategic learning?
Distributed strategic learning is learning in the presence of other players (i.e., from, with, or about others) or (from, with or about) Nature/ environment.
The term strategic learning stands for how the players can learn from/with/about the others (players, environment) under their complex strategies/beliefs.

41. Strategic Mean Field Learning
The term mean field learning refers to a learning framework for large populations of players, which is highly related to HJB and/ or FPK equations.
Basically, we summarize that there exist two kinds of mean field learning techniques:
FPK forward equation, and mean field learning techniques
HJB backward equation, which relates to the Hamiltonian.

42. Strategic Mean Field Learning

43. Strategic Mean Field Learning
Huibing Yin; Mehta, P.G.; Meyn, S.P.; Shanbhag, U.V., "Learning in Mean-Field Games," in Automatic Control, IEEE Transactions on , vol.59, no.3, pp , March 2014
Mehta, P.; Meyn, S., "Q-learning and Pontryagin's Minimum Principle," in Decision and Control, 2009 held jointly with the th Chinese Control Conference. CDC/CCC Proceedings of the 48th IEEE Conference on , vol., no., pp , Dec. 2009

44. Future Work Working on:
Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator
Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective
Advanced MFG Framework:
Robust MFG
Cooperative MFG ……
New Applications: IoT, Cloud, and Smart Gird; Various Techniques.
Cooperative Spectrum Leasing in LTE Un-licensed Spectrum: A Mean Field Game
Model the spectrum competition behaviors of denser deployment of small cells as mean field, then how to determine the spectrum leasing, offloading?
Spectrum pricing?
Hybrid MFG with dominant player?
Social-Aware D2D Communication Networks: Social Behavior
Community-based
New applications in smart grid, and how the community behavior affects the decision? How to use the power?

45. Conclusion
Introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. After that, we introduce the basics of MFG framework. 
Present  a power control MFG in the ultra dense D2D networks, where our most important contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player.
Present a interference mitigation MFG with interference dominator. Consider interactive power control policies for both the dominator (Major player) and generic players (Minor players) in heterogeneous small cell UDNs.
Identify relation between the Q-value and Hamiltonian. Thus, we turn to strategic MFG learning design. 
Look forward our future work, and then conclude the presentation.
In this presentation, we first introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. Then, we introduce the basics of MFG. Following that, we present our works in the MFG framework. First, we present a power control MFG in the ultra-dense D2D networks, where our contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player. Second, we present a interference mitigation MFG with interference dominator. It is the first to jointly consider the interactive power control policies for both the dominator and generic players in heterogeneous small cell UDNs. Third, MFG is hard to solve, mainly due to Hamiltonian. However, there is natural relation between the Q-value and Hamiltonian. Thus, we turn strategic MFG learning design. Finally, we look forward our future work, and conclude the presentation.

46. Questions?