Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 14 Mean-Field Game Zhu Han, Dusit Niyato, Walid Saad, and Tamer Basar Thanks for Dr. Chungang Yang slides

Overview of Lecture Notes Introduction to Game Theory: Lecture 1, book 1 Non-cooperative Games: Lecture 1, Chapter 3, book 1 Bayesian Games: Lecture 2, Chapter 4, book 1 Differential Games: Lecture 3, Chapter 5, book 1 Evolutionary Games: Lecture 4, Chapter 6, book 1 Cooperative Games: Lecture 5, Chapter 7, book 1 Auction Theory: Lecture 6, Chapter 8, book 1 Matching Game: Lecture 7, Chapter 2, book 2 Contract Theory, Lecture 8, Chapter 3, book 2 Learning in Game, Lecture 9, Chapter 6, book 2 Stochastic Game, Lecture 10, Chapter 4, book 2 Game with Bounded Rationality, Lecture 11, Chapter 5, book 2 Equilibrium Programming with Equilibrium Constraint, Lecture 12, Chapter 7, book 2 Zero Determinant Strategy, Lecture 13, Chapter 8, book 2 Mean Field Game, Lecture 14, UCLA course Network Economy, Lecture 15, Dr. Jianwei Huang,

Outline Basics of MFG Applications of MFG Conventional MFG in Ultra-Dense D2D Communications Ultra-Dense Small Cell Networks: MFG with Interference Dominator Mean Field Game-Theoretic Learning Conclusion What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need? What is MFG? Applications of MFG in networks, and in particular we

Mean Field Game, MFG Basic Idea: While classical game theory models the interaction of a single player with all the other players of the system, a mean field game models the individual's interaction with the effect of the collective behavior of the players. Think globally while behave locally Think individually while behave locally What is MFG? Why MFG? Basically, MFG can be considered a special form of differential games with a large number of players. Here, we know that conventional game models the interaction of one rational player with all the other players, however, the MFG models the any individual’s selfish behavior with respect to the collective behavior of all the other players. Here, the collective behavior is named as a mass.

Mean Field Game, MFG Understand rational behaviors, and optimize their position in space and time dynamics. The mean field theory is exploited to help decouple a complex large-scale optimization problem into a family of localized optimization problems. Thus, each player can implement its policy by using only its local information.

Mean Field Game, MFG Mean Field Game describes equilibrium configurations and dynamic behaviors of huge number of homogeneous agents, which is a class of differential games in which each agent is infinitesimal and interacts with a huge population of other agents. Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, x) is the value function of a typical small player while m = m(t, x) denotes the mean field value at time t and at position x of the population. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

Mean Field Game, MFG The value function u satisfies a Hamilton-Jacobi-Bellman (HJB) equation—in which m enters as a parameter and describes the influence of the population on the cost of each agent, while the density m evolves in time according to a Fokker-Planck-Kolmogorov (FPK) equation in which u enters as a drift. More precisely the pair (u,m) is a solution of the MFG system. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

Mean Field Game, MFG HJB equation governs the computation of the optimal path of control of the player, while FPK equation governs the evolution of the mean field function of players. Here, the HJB and FPK equations are termed as the backward and forward functions, respectively. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent. Combination of them in time creates some unusual phenomena in the time variable .

Outline Basics of MFG Applications of MFG Conventional MFG in Ultra-Dense D2D Communications Ultra-Dense Small Cell Networks: MFG with Interference Dominator Mean Field Game-Theoretic Learning Conclusion What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need? What is MFG? Applications of MFG in networks, and in particular we

Challenges in 5G Ultra-Dense Networks IoE: Internet of Everything Mobile Internet 5G Ultra-Dense Small Cells Social-Aware D2D Networks Internet of Things M2M, V2V Networks 5G Characteristics: Ultra-Dense, Scalable, Self-Organizing; Energy efficiency, spectral efficiency, cost efficiency In 5G era, everything will be connected. It is called IoE for short. On the other hand, for better QoE of users, ultra-dense small cells and D2D networks, IOT creating opportunities for more direct integration between the physical world and computer-based systems, and resulting in improved efficiency, accuracy and economic benefit. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications (M2M) and covers a variety of protocols, domains, and applications.[10] The interconnection of these embedded devices (including smart objects), is expected to usher in automation in nearly all fields, while also enabling advanced applications like a Smart Grid,[11] and expanding to the areas such as Smart city.

Challenges in 5G Ultra-Dense Networks Technical Challenges Coupled Systems & Interference -Limited Resources Computationally Intractable & Analysis –Number Infinity Low Convergence Rate & Robustness -Dynamics Perfect Information & Backhauling Overhead -Others Requirements Self-Organization & Distributed Control Space-time Dynamics

Game Theory in 5G Era Extensive applications to characterize the rational behaviors, model the strategic interactions, and design distributed algorithms. Face great challenges, e.g., the number of the nodes, e.g., access points and small cell base stations, is huge, even goes to infinity, leading to the well-known curse of dimensionality. Hard to analyze the game due to the involved dynamics, complexity, interaction, signaling overhead, and so on.

Applications of MFG in Wireless Networks Green Communications and Cognitive Radio Networks H. Tembine, R. Tempone, and R. Vilanova, "Mean field games for cognitive radio networks", American Control Conference (ACC), Pages: 6388-6393, 2012. F. Meriaux and S. Lasaulce, "Mean-field games and green power control", 2011 5th International Conference on Network Games, Control and Optimization (NetGCooP), Pages: 1-5, 2011. Hyper-Dense Heterogeneous Small Cell Networks Prabodini Semasinghe and Ekram Hossain, Downlink Power Control in Self-Organizing DenseSmall Cells Underlaying Macrocells: A Mean Field Game, IEEE Transactions on Mobile Computing. Ali Y. Al-Zahrani, F. Richard Yu, and Minyi Huang, A Joint Cross-Layer and Co-Layer Interference Management Scheme in Hyper-Dense Heterogeneous Networks Using Mean-Field Game Theory, IEEE Transactions on Vehicular Technology, 2015. Cloud and Smart V2V Networks A. F. Hanif, H. Tembine, M. Assaad, and D. Zeghlache, "Mean-field games for resource sharing in cloud-based networks", IEEE/ACMTransactions on Networking, 2015. R. Couillet, S.M. Perlaza, H. Tembine, and M. Debbah, "Electrical Vehicles in the Smart Grid-A Mean Field Game Analysis", IEEE Journal on Selected Areas in Communications, Publication Year: 2012 , Page(s): 1086 - 1096.

Distributed Interference and Energy Aware Power Control for Ultra-Dense D2D Networks: A Mean Field Game

Ultra-Dense D2D Networks Device-to-device (D2D) provides significant performance enhancement in terms of spectrum and energy efficiency by proximity and frequency reuse. However, such performance enhancement is largely limited by mutual interference and energy availability, in particular, in ultra-dense D2D networks. Powered by battery. Thus, an intelligent power control remaining energy of the battery.

Ultra-Dense D2D Networks Interference dynamics and available energy, and then formulate a MFG theoretic framework with the interference mean-field approximation.. Investigate effects of both the interference dynamics of the generic device introduced to others, and all others' interference dynamics introduced into the generic device on power control.

D2D Differential Power Control Game Remaining Energy Interference Dynamics Interference dynamics of the generic device introduced to others

D2D Differential Power Control Game All others' interference dynamics introduced into the generic device Instantaneous Cost function at time t, which is quadratic function of power consumption

Problem Formulation Optimal control problem followed by Bellman's optimality principle. Control Policy Instantaneous Cost Remaining Energy State Space Interference Dynamics

Solution Hamilton-Jacobi-Bellman (HJB) : a partial differential equation…central to optimal control theory. Solution of HJB is the value function, which gives the minimum cost for a given dynamical system with an associated cost function. Value function describes the minimum total cost from time t to time t+dt using specific policy. Value function at time t+dt. Cost function is determined by state and power at time t

Solution Hamilton-Jacobi-Bellman (HJB) :

Mean-Field Game Obtaining equilibrium for a system with N players involves solving N simultaneous partial differential equations. State Dynamics and Number of Players Mean field Fokker-Planck-Kolmogorov (FPK) equation Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players involves solving $N$ simultaneous partial differential equations.

Mean-Field Game Mean field & FPK Equation Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players involves solving $N$ simultaneous partial differential equations.

MFG-Based Analysis HJB Backward FPK Forward Hamiltonian Results Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, s) is the value function of a typical small player while m = m(t, s) denotes the mean field value at time t and at state s of the population.

MFG-Based Algorithm Finite difference Technique in discretized way of space Backward Forward

Simulation Results Extended Battery Life and Improved Energy Efficiency

Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator

Ultra-Dense Small Cell Networks Extensive deployment of various small cells underlaying cellular networks reuses of the limited frequency resource to improve spectrum efficiency, thus interference. Number tends to infinity, centralized interference management encounters signaling overhead, scalability, and flexibility, mainly due to random deployments and limited backhaul. Therefore, decentralized interference management coupled with self-organizing properties should be studied, and that is why game theory has found extensive applications recently.

Ultra-Dense Small Cell Networks Here, we choose SeNB1 as the generic player, and it will receive the both inter-tier interference and intra-tier interference. Here, we know the downlink power of the MeNB is stronger than that of the SeNBs, thus resulting MeNB as the interference dominator of the SeNB1 and other SeNBs as the minor interferers.

Mean Field Game with Interference Dominator We jointly take the global and individual perspective to react to the interference when the number of small cell players tends to infinity. To analyze and characterize the equilibrium behaviors, we formulate it as a mean field game, which simplify the analysis and facilitate distributed interference mitigation. To think globally while behave locally, in practice an interference dominator always exists in addition to the mean field term, which would significantly affects the decision-making for the representative agent. Therefore, we turn to mean field games between a dominating player and a group of representative agents, each of which acts similarly and also interacts with each other through a mean field term being substantially influenced by the interference dominating player.

Interference Mitigation Stochastic Game With tow kinds of players: Interference dominator and minor interferers Interference perceived by the interference dominator Interference perceived by the minor generic interferer Interference mean field:

Interference Mitigation Stochastic Game

Interference Mitigation Stochastic Game

MFG with Interference Dominator We derive the HJB equation We derive the FPK equation

Solution Mean Field Update

Solution Introduce Lagrangian Parameter to indirectly solve the HJB

Solution Power Update

Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective

Strategic Mean Field Learning What is distributed strategic learning? Distributed strategic learning is learning in the presence of other players (i.e., from, with, or about others) or (from, with or about) Nature/ environment. The term strategic learning stands for how the players can learn from/with/about the others (players, environment) under their complex strategies/beliefs.

Strategic Mean Field Learning The term mean field learning refers to a learning framework for large populations of players, which is highly related to HJB and/ or FPK equations. Basically, we summarize that there exist two kinds of mean field learning techniques: FPK forward equation, and mean field learning techniques HJB backward equation, which relates to the Hamiltonian.

Strategic Mean Field Learning

Strategic Mean Field Learning Huibing Yin; Mehta, P.G.; Meyn, S.P.; Shanbhag, U.V., "Learning in Mean-Field Games," in Automatic Control, IEEE Transactions on , vol.59, no.3, pp.629-644, March 2014 Mehta, P.; Meyn, S., "Q-learning and Pontryagin's Minimum Principle," in Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on , vol., no., pp.3598-3605, 15-18 Dec. 2009

Future Work Working on: Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective Advanced MFG Framework: Robust MFG Cooperative MFG …… New Applications: IoT, Cloud, and Smart Gird; Various Techniques. Cooperative Spectrum Leasing in LTE Un-licensed Spectrum: A Mean Field Game Model the spectrum competition behaviors of denser deployment of small cells as mean field, then how to determine the spectrum leasing, offloading? Spectrum pricing? Hybrid MFG with dominant player? Social-Aware D2D Communication Networks: Social Behavior Community-based New applications in smart grid, and how the community behavior affects the decision? How to use the power?

Conclusion Introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. After that, we introduce the basics of MFG framework.  Present  a power control MFG in the ultra dense D2D networks, where our most important contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player. Present a interference mitigation MFG with interference dominator. Consider interactive power control policies for both the dominator (Major player) and generic players (Minor players) in heterogeneous small cell UDNs. Identify relation between the Q-value and Hamiltonian. Thus, we turn to strategic MFG learning design.  Look forward our future work, and then conclude the presentation. In this presentation, we first introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. Then, we introduce the basics of MFG. Following that, we present our works in the MFG framework. First, we present a power control MFG in the ultra-dense D2D networks, where our contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player. Second, we present a interference mitigation MFG with interference dominator. It is the first to jointly consider the interactive power control policies for both the dominator and generic players in heterogeneous small cell UDNs. Third, MFG is hard to solve, mainly due to Hamiltonian. However, there is natural relation between the Q-value and Hamiltonian. Thus, we turn strategic MFG learning design. Finally, we look forward our future work, and conclude the presentation.

Questions?