Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 14 Mean-Field Game Zhu Han, Dusit Niyato, Walid Saad, and.

Slides:



Advertisements
Similar presentations
Continuation Methods for Structured Games Ben Blum Christian Shelton Daphne Koller Stanford University.
Advertisements

End-to-End Efficiency (E 3 ) Integrated Project of the EC 7 th Framework Programme E 3 WP5 Objectives E 3 WP5 Structure and Research Challenges
Sogang University ICC Lab Using Game Theory to Analyze Wireless Ad Hoc networks.
Game-Theoretic Approaches to Multi-Agent Systems Bernhard Nebel.
1 Sensor Networks and Networked Societies of Artifacts Jose Rolim University of Geneva.
Dynamic Spectrum Management: Optimization, game and equilibrium Tom Luo (Yinyu Ye) December 18, WINE 2008.
1 Cross-Layer Design for Wireless Communication Networks Ness B. Shroff Center for Wireless Systems and Applications (CWSA) School of Electrical and Computer.
Nash Q-Learning for General-Sum Stochastic Games Hu & Wellman March 6 th, 2006 CS286r Presented by Ilan Lobel.
1 Hybrid Agent-Based Modeling: Architectures,Analyses and Applications (Stage One) Li, Hailin.
D Nagesh Kumar, IIScOptimization Methods: M1L4 1 Introduction and Basic Concepts Classical and Advanced Techniques for Optimization.
Scheduling of Wireless Metering for Power Market Pricing in Smart Grid Husheng Li, Lifeng Lai, and Robert Caiming Qiu. "Scheduling of Wireless Metering.
Energy arbitrage with micro-storage UKACC PhD Presentation Showcase Antonio De Paola Supervisors: Dr. David Angeli / Prof. Goran Strbac Imperial College.
COGNITIVE RADIO FOR NEXT-GENERATION WIRELESS NETWORKS: AN APPROACH TO OPPORTUNISTIC CHANNEL SELECTION IN IEEE BASED WIRELESS MESH Dusit Niyato,
1 IEEE Trans. on Smart Grid, 3(1), pp , Optimal Power Allocation Under Communication Network Externalities --M.G. Kallitsis, G. Michailidis.
A Framework for Distributed Model Predictive Control
Wireless Networks Breakout Session Summary September 21, 2012.
1 Performance Analysis of Coexisting Secondary Users in Heterogeneous Cognitive Radio Network Xiaohua Li Dept. of Electrical & Computer Engineering State.
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 3 Differential Game Zhu Han, Dusit Niyato, Walid Saad, Tamer.
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 4 Evolutional Game Zhu Han, Dusit Niyato, Walid Saad, Tamer.
Secondary Pricing of Spectrum in Cellular CDMA Networks Ashraf Al Daoud, Murat Alanyali, and David Starobinski Department of Electrical and Computer Engineering.
Advanced Spectrum Management in Multicell OFDMA Networks enabling Cognitive Radio Usage F. Bernardo, J. Pérez-Romero, O. Sallent, R. Agustí Radio Communications.
Downlink Scheduling With Economic Considerations to Future Wireless Networks Bader Al-Manthari, Nidal Nasser, and Hossam Hassanein IEEE Transactions on.
COGNITIVE RADIO NETWORKING AND RENDEZVOUS Presented by Vinay chekuri.
Multi-Radio Integration for Heterogeneous IEEE Network Beyond 4G IEEE Presentation Submission Template (Rev. 9) Document Number: IEEE C /0015.
Information Theory for Mobile Ad-Hoc Networks (ITMANET): The FLoWS Project Competitive Scheduling in Wireless Networks with Correlated Channel State Ozan.
Designing Games for Distributed Optimization Na Li and Jason R. Marden IEEE Journal of Selected Topics in Signal Processing, Vol. 7, No. 2, pp ,
Efficient Resource Allocation for Wireless Multicast De-Nian Yang, Member, IEEE Ming-Syan Chen, Fellow, IEEE IEEE Transactions on Mobile Computing, April.
Status & Challenges Interoperability and global integration of communication infrastructure & service platform Fixed-mobile convergence to achieve a future.
Smart Sleeping Policies for Wireless Sensor Networks Venu Veeravalli ECE Department & Coordinated Science Lab University of Illinois at Urbana-Champaign.
1 Architecture and Behavioral Model for Future Cognitive Heterogeneous Networks Advisor: Wei-Yeh Chen Student: Long-Chong Hung G. Chen, Y. Zhang, M. Song,
Asymptotic Analysis for Large Scale Dynamic Stochastic Games Sachin Adlakha, Ramesh Johari, Gabriel Weintraub and Andrea Goldsmith DARPA ITMANET Meeting.
Submission May 2016 H. H. LEESlide 1 IEEE Framework and Its Applicability to IMT-2020 Date: Authors:
Toward An Understanding of Self-Organization of Markets Yougui Wang Department of Systems Science, School of Management, Beijing Normal University, Beijing.
Medium Access Control. MAC layer covers three functional areas: reliable data delivery access control security.
Mohsen Riahi Manesh and Dr. Naima Kaabouch
Younes Abdi, PhD Faculty of Information Technology
Satisfaction Games in Graphical Multi-resource Allocation
Lecture 7: Internet of Things
Distributed Mobility Management for Future 5G Networks : Overview and Analysis of Existing Approaches IEEE Wireless Communications January 2015 F. Giust,
Cognitive Radio Networks
On Routine Evolution of Complex Cellular Automata
The 2st Chinese Workshop on Evolutionary Computation and Learning
Near-Optimal Spectrum Allocation for Cognitive Radios: A Frequency-Time Auction Perspective Xinyu Wang Department of Electronic Engineering Shanghai.
Cellular IP: A New Approach to Internet Host Mobility
Ad-hoc Networks.
Zhu Han University of Houston Thanks for Dr. Mingyi Hong’s slides
UNIT II –Part 2.
Q-Learning for Policy Improvement in Network Routing
Energy Efficiency in HEW
Algorithms for Big Data Delivery over the Internet of Things
User Interference Effect on Routing of Cognitive Radio Ad-Hoc Networks
Environment-Aware Reputation Management for Ad Hoc Networks
When Security Games Go Green
به نام خدا Big Data and a New Look at Communication Networks Babak Khalaj Sharif University of Technology Department of Electrical Engineering.
Zhu Han, Dusit Niyato, Walid Saad, and Tamer Basar
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 6 Auction Theory Zhu Han, Dusit Niyato, Walid Saad, Tamer.
Artificial Intelligence
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 2 Bayesian Games Zhu Han, Dusit Niyato, Walid Saad, Tamer.
I-Kang Fu, Paul Cheng, MediaTek
Zero Determinant Strategy
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 10 Stochastic Game Zhu Han, Dusit Niyato, Walid Saad, and.
Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 11 Game with Bounded Rationality Zhu Han, Dusit Niyato, Walid.
Artificial Intelligence
Consideration on applying ICN to Edge Computing
Market-based Dynamic Task Allocation in Mobile Surveillance Systems
Spectrum Sharing in Cognitive Radio Networks
Oblivious Equilibrium for Stochastic Games with Concave Utility
Speaker: Ao Weng Chon Advisor: Kwang-Cheng Chen
HUAWEI Technology 2019/8/25 Cognitive Radio Networks: Imagination or Reality? CrownCom 2008 HUAWEI Technology
Feasibility of Coordinated Transmission for HEW
Presentation transcript:

Game Theory in Wireless and Communication Networks: Theory, Models, and Applications Lecture 14 Mean-Field Game Zhu Han, Dusit Niyato, Walid Saad, and Tamer Basar Thanks for Dr. Chungang Yang slides

Overview of Lecture Notes Introduction to Game Theory: Lecture 1, book 1 Non-cooperative Games: Lecture 1, Chapter 3, book 1 Bayesian Games: Lecture 2, Chapter 4, book 1 Differential Games: Lecture 3, Chapter 5, book 1 Evolutionary Games: Lecture 4, Chapter 6, book 1 Cooperative Games: Lecture 5, Chapter 7, book 1 Auction Theory: Lecture 6, Chapter 8, book 1 Matching Game: Lecture 7, Chapter 2, book 2 Contract Theory, Lecture 8, Chapter 3, book 2 Learning in Game, Lecture 9, Chapter 6, book 2 Stochastic Game, Lecture 10, Chapter 4, book 2 Game with Bounded Rationality, Lecture 11, Chapter 5, book 2 Equilibrium Programming with Equilibrium Constraint, Lecture 12, Chapter 7, book 2 Zero Determinant Strategy, Lecture 13, Chapter 8, book 2 Mean Field Game, Lecture 14, UCLA course Network Economy, Lecture 15, Dr. Jianwei Huang,

Outline Basics of MFG Applications of MFG Conventional MFG in Ultra-Dense D2D Communications Ultra-Dense Small Cell Networks: MFG with Interference Dominator Mean Field Game-Theoretic Learning Conclusion What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need? What is MFG? Applications of MFG in networks, and in particular we

Mean Field Game, MFG Basic Idea: While classical game theory models the interaction of a single player with all the other players of the system, a mean field game models the individual's interaction with the effect of the collective behavior of the players. Think globally while behave locally Think individually while behave locally What is MFG? Why MFG? Basically, MFG can be considered a special form of differential games with a large number of players. Here, we know that conventional game models the interaction of one rational player with all the other players, however, the MFG models the any individual’s selfish behavior with respect to the collective behavior of all the other players. Here, the collective behavior is named as a mass.

Mean Field Game, MFG Understand rational behaviors, and optimize their position in space and time dynamics. The mean field theory is exploited to help decouple a complex large-scale optimization problem into a family of localized optimization problems. Thus, each player can implement its policy by using only its local information.

Mean Field Game, MFG Mean Field Game describes equilibrium configurations and dynamic behaviors of huge number of homogeneous agents, which is a class of differential games in which each agent is infinitesimal and interacts with a huge population of other agents. Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, x) is the value function of a typical small player while m = m(t, x) denotes the mean field value at time t and at position x of the population. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

Mean Field Game, MFG The value function u satisfies a Hamilton-Jacobi-Bellman (HJB) equation—in which m enters as a parameter and describes the influence of the population on the cost of each agent, while the density m evolves in time according to a Fokker-Planck-Kolmogorov (FPK) equation in which u enters as a drift. More precisely the pair (u,m) is a solution of the MFG system. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent.

Mean Field Game, MFG HJB equation governs the computation of the optimal path of control of the player, while FPK equation governs the evolution of the mean field function of players. Here, the HJB and FPK equations are termed as the backward and forward functions, respectively. Fokker-Plank: evolving forward in time that governs the evolution of the mass function of the system; Hamilton-Jacobi-Bellman: evolving backward in time that governs the computation of the optimal path for each agent. Combination of them in time creates some unusual phenomena in the time variable .

Outline Basics of MFG Applications of MFG Conventional MFG in Ultra-Dense D2D Communications Ultra-Dense Small Cell Networks: MFG with Interference Dominator Mean Field Game-Theoretic Learning Conclusion What is 5G ultra-dense networks; what are the critical technical characteristics? What new game model we need? What is MFG? Applications of MFG in networks, and in particular we

Challenges in 5G Ultra-Dense Networks IoE: Internet of Everything Mobile Internet 5G Ultra-Dense Small Cells Social-Aware D2D Networks Internet of Things M2M, V2V Networks 5G Characteristics: Ultra-Dense, Scalable, Self-Organizing; Energy efficiency, spectral efficiency, cost efficiency In 5G era, everything will be connected. It is called IoE for short. On the other hand, for better QoE of users, ultra-dense small cells and D2D networks, IOT creating opportunities for more direct integration between the physical world and computer-based systems, and resulting in improved efficiency, accuracy and economic benefit. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications (M2M) and covers a variety of protocols, domains, and applications.[10] The interconnection of these embedded devices (including smart objects), is expected to usher in automation in nearly all fields, while also enabling advanced applications like a Smart Grid,[11] and expanding to the areas such as Smart city.

Challenges in 5G Ultra-Dense Networks Technical Challenges Coupled Systems & Interference -Limited Resources Computationally Intractable & Analysis –Number Infinity Low Convergence Rate & Robustness -Dynamics Perfect Information & Backhauling Overhead -Others Requirements Self-Organization & Distributed Control Space-time Dynamics

Game Theory in 5G Era Extensive applications to characterize the rational behaviors, model the strategic interactions, and design distributed algorithms. Face great challenges, e.g., the number of the nodes, e.g., access points and small cell base stations, is huge, even goes to infinity, leading to the well-known curse of dimensionality. Hard to analyze the game due to the involved dynamics, complexity, interaction, signaling overhead, and so on.

Applications of MFG in Wireless Networks Green Communications and Cognitive Radio Networks H. Tembine, R. Tempone, and R. Vilanova, "Mean field games for cognitive radio networks", American Control Conference (ACC), Pages: 6388-6393, 2012. F. Meriaux and S. Lasaulce, "Mean-field games and green power control", 2011 5th International Conference on Network Games, Control and Optimization (NetGCooP), Pages: 1-5, 2011. Hyper-Dense Heterogeneous Small Cell Networks Prabodini Semasinghe and Ekram Hossain, Downlink Power Control in Self-Organizing DenseSmall Cells Underlaying Macrocells: A Mean Field Game, IEEE Transactions on Mobile Computing. Ali Y. Al-Zahrani, F. Richard Yu, and Minyi Huang, A Joint Cross-Layer and Co-Layer Interference Management Scheme in Hyper-Dense Heterogeneous Networks Using Mean-Field Game Theory, IEEE Transactions on Vehicular Technology, 2015. Cloud and Smart V2V Networks A. F. Hanif, H. Tembine, M. Assaad, and D. Zeghlache, "Mean-field games for resource sharing in cloud-based networks", IEEE/ACMTransactions on Networking, 2015. R. Couillet, S.M. Perlaza, H. Tembine, and M. Debbah, "Electrical Vehicles in the Smart Grid-A Mean Field Game Analysis", IEEE Journal on Selected Areas in Communications, Publication Year: 2012 , Page(s): 1086 - 1096.

Distributed Interference and Energy Aware Power Control for Ultra-Dense D2D Networks: A Mean Field Game

Ultra-Dense D2D Networks Device-to-device (D2D) provides significant performance enhancement in terms of spectrum and energy efficiency by proximity and frequency reuse. However, such performance enhancement is largely limited by mutual interference and energy availability, in particular, in ultra-dense D2D networks. Powered by battery. Thus, an intelligent power control remaining energy of the battery.

Ultra-Dense D2D Networks Interference dynamics and available energy, and then formulate a MFG theoretic framework with the interference mean-field approximation.. Investigate effects of both the interference dynamics of the generic device introduced to others, and all others' interference dynamics introduced into the generic device on power control.

D2D Differential Power Control Game Remaining Energy Interference Dynamics Interference dynamics of the generic device introduced to others

D2D Differential Power Control Game All others' interference dynamics introduced into the generic device Instantaneous Cost function at time t, which is quadratic function of power consumption

Problem Formulation Optimal control problem followed by Bellman's optimality principle. Control Policy Instantaneous Cost Remaining Energy State Space Interference Dynamics

Solution Hamilton-Jacobi-Bellman (HJB) : a partial differential equation…central to optimal control theory. Solution of HJB is the value function, which gives the minimum cost for a given dynamical system with an associated cost function. Value function describes the minimum total cost from time t to time t+dt using specific policy. Value function at time t+dt. Cost function is determined by state and power at time t

Solution Hamilton-Jacobi-Bellman (HJB) :

Mean-Field Game Obtaining equilibrium for a system with N players involves solving N simultaneous partial differential equations. State Dynamics and Number of Players Mean field Fokker-Planck-Kolmogorov (FPK) equation Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players involves solving $N$ simultaneous partial differential equations.

Mean-Field Game Mean field & FPK Equation Obtaining the equilibrium for game $G_{s}$ for a system with $N$ players involves solving $N$ simultaneous partial differential equations.

MFG-Based Analysis HJB Backward FPK Forward Hamiltonian Results Solution in Mean Field Game is given by a pair of maps (u,m), where u= u(t, s) is the value function of a typical small player while m = m(t, s) denotes the mean field value at time t and at state s of the population.

MFG-Based Algorithm Finite difference Technique in discretized way of space Backward Forward

Simulation Results Extended Battery Life and Improved Energy Efficiency

Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator

Ultra-Dense Small Cell Networks Extensive deployment of various small cells underlaying cellular networks reuses of the limited frequency resource to improve spectrum efficiency, thus interference. Number tends to infinity, centralized interference management encounters signaling overhead, scalability, and flexibility, mainly due to random deployments and limited backhaul. Therefore, decentralized interference management coupled with self-organizing properties should be studied, and that is why game theory has found extensive applications recently.

Ultra-Dense Small Cell Networks Here, we choose SeNB1 as the generic player, and it will receive the both inter-tier interference and intra-tier interference. Here, we know the downlink power of the MeNB is stronger than that of the SeNBs, thus resulting MeNB as the interference dominator of the SeNB1 and other SeNBs as the minor interferers.

Mean Field Game with Interference Dominator We jointly take the global and individual perspective to react to the interference when the number of small cell players tends to infinity. To analyze and characterize the equilibrium behaviors, we formulate it as a mean field game, which simplify the analysis and facilitate distributed interference mitigation. To think globally while behave locally, in practice an interference dominator always exists in addition to the mean field term, which would significantly affects the decision-making for the representative agent. Therefore, we turn to mean field games between a dominating player and a group of representative agents, each of which acts similarly and also interacts with each other through a mean field term being substantially influenced by the interference dominating player.

Interference Mitigation Stochastic Game With tow kinds of players: Interference dominator and minor interferers Interference perceived by the interference dominator Interference perceived by the minor generic interferer Interference mean field:

Interference Mitigation Stochastic Game

Interference Mitigation Stochastic Game

MFG with Interference Dominator We derive the HJB equation We derive the FPK equation

Solution Mean Field Update

Solution Introduce Lagrangian Parameter to indirectly solve the HJB

Solution Power Update

Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective

Strategic Mean Field Learning What is distributed strategic learning? Distributed strategic learning is learning in the presence of other players (i.e., from, with, or about others) or (from, with or about) Nature/ environment. The term strategic learning stands for how the players can learn from/with/about the others (players, environment) under their complex strategies/beliefs.

Strategic Mean Field Learning The term mean field learning refers to a learning framework for large populations of players, which is highly related to HJB and/ or FPK equations. Basically, we summarize that there exist two kinds of mean field learning techniques: FPK forward equation, and mean field learning techniques HJB backward equation, which relates to the Hamiltonian.

Strategic Mean Field Learning

Strategic Mean Field Learning Huibing Yin; Mehta, P.G.; Meyn, S.P.; Shanbhag, U.V., "Learning in Mean-Field Games," in Automatic Control, IEEE Transactions on , vol.59, no.3, pp.629-644, March 2014 Mehta, P.; Meyn, S., "Q-learning and Pontryagin's Minimum Principle," in Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on , vol., no., pp.3598-3605, 15-18 Dec. 2009

Future Work Working on: Distributed Interference Mitigation in Ultra-Dense Small Cell Networks: Mean Field Game with Interference Dominator Self-Confirming Interference Management Policy in 5G Ultra-Dense Small Cell Networks: A Mean Field Game-Theoretic Learning Perspective Advanced MFG Framework: Robust MFG Cooperative MFG …… New Applications: IoT, Cloud, and Smart Gird; Various Techniques. Cooperative Spectrum Leasing in LTE Un-licensed Spectrum: A Mean Field Game Model the spectrum competition behaviors of denser deployment of small cells as mean field, then how to determine the spectrum leasing, offloading? Spectrum pricing? Hybrid MFG with dominant player? Social-Aware D2D Communication Networks: Social Behavior Community-based New applications in smart grid, and how the community behavior affects the decision? How to use the power?

Conclusion Introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. After that, we introduce the basics of MFG framework.  Present  a power control MFG in the ultra dense D2D networks, where our most important contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player. Present a interference mitigation MFG with interference dominator. Consider interactive power control policies for both the dominator (Major player) and generic players (Minor players) in heterogeneous small cell UDNs. Identify relation between the Q-value and Hamiltonian. Thus, we turn to strategic MFG learning design.  Look forward our future work, and then conclude the presentation. In this presentation, we first introduce the technical challenges in 5G UDNs, which is the motivation of why we need MFG. Then, we introduce the basics of MFG. Following that, we present our works in the MFG framework. First, we present a power control MFG in the ultra-dense D2D networks, where our contribution is we jointly investigated the impacts of interference dynamics and remaining energy on the power control of a generic D2D player. Second, we present a interference mitigation MFG with interference dominator. It is the first to jointly consider the interactive power control policies for both the dominator and generic players in heterogeneous small cell UDNs. Third, MFG is hard to solve, mainly due to Hamiltonian. However, there is natural relation between the Q-value and Hamiltonian. Thus, we turn strategic MFG learning design. Finally, we look forward our future work, and conclude the presentation.

Questions?