1. Multiple-access Communication in Networks A Geometric View W. Chen & S. Meyn Dept ECE & CSL University of Illinois

2. Implementation – Consensus algorithms & Information distribution Adaptation – Reinforcement learning techniques Integration with Network Coding projects: Code around network hot-spots What is the state of the art and what are its limitations? Notes from Austin: MW routing inflexible, and does not easily incorporate multi- access capacity region in wireless. Workload relaxation techniques: Tremendous value for policy synthesis based on dynamic hot- spots in the network Can these techniques be extended to wireless models? Relaxation Techniques for Net Opt W. Chen & S. Meyn KEY NEW INSIGHTS: Extend to wireless? YES Geometric picture is very different. Interpretation: The number of resources is infinite Structure of optimal solution to relaxation is very simple, even for very complex networks New application of relaxation: Q-learning and TD-learning for routing and power control Un-consummated union challenge: Integrate coding and resource allocation Generally, solutions to complex decision problems should offer insight Algorithms for dynamic routing: Visualization and Optimization END-OF-PHASE GOAL COMMUNITY CHALLENGE ACHIEVEMENT DESCRIPTION STATUS QUO NEW INSIGHTS MAIN RESULT: HOW IT WORKS: Step 1: Estimate capacity region near estimated allocation rate vector Step 2: Construct half-space relaxation Step 3: Optimal policy for relaxation: Buffer priorities, based on coefficients of normal vector Numerical findings: With many flows, the rate region appears smooth even in a static interference model Dynamics of 720 queues Half space relaxation provides: Impact: Network cut is no longer a useful concept Lower bound on performance and Tools for policy synthesis Infinite complexity leads to simple solution:

3. Where to focus attention for coding and routing? Understanding MANETs – where do we direct genius for coding and control? Issues addressed: Where should effort be directed in coding and control for complex networks? Performance evaluation: Lower bounds, and approximate optimality Special attention to issues surrounding MANETs: Multiple access phenomena and fading Message from 15 years of research: Achieving stability is possible using very simple routing schemes. Implementation in multiple access settings possible with a bit of genius Lacking: Methods to improve delay performance, and methods to make appropriate tradeoffs between throughput and delay.

4. Decision & Control Perspective Understanding MANETs – Relaxations capture essential constraints and dynamics D&C Perspective for networks 1.Simple, idealized model is the dynamic fluid model 2.Further simplification to obtain the workload relaxation 3.Lower bounds on performance, and control solutions from the relaxation 4.Lyapunov based design constitutes translation to network (e.g. h-MaxWeight). D&C Perspective: Obtain the simplest model that captures essential constraints and dynamics Design highly robust control solution for the simple model Translate design to the relatively complex system

5. Example: Dynamic Power Control Power control solved using fluid model + reinforcement learning techniques Dynamic speed scaling 1. Model: Fluid and stochastic model for arrivals, and controlled service rate 2. Further relaxation not required in single link model 3. Solve DP equation for fluid model 4. Solution to fluid model used to construct architecture for reinforcement learning Approximate Dynamic Programming using Fluid and Diffusion Approximations with Applications to Power Management Input power as function of queue length. Policy for fluid model closely approximates the optimal average delay policy for the discrete/stochastic model Optimal control solution for fluid model gives perfect architecture for on-line learning/on-line optimization

6. Example: h-MaxWeight Policy Optimal control of complex routing models solved using workload relaxation h-MaxWeight policy - introduced in ITMANET project 1. Model: Fluid and stochastic model for arrivals, and controlled service rate 2. Workload relaxations – dynamic generalization of network cuts 3. Optimal control for relaxation is simple 4. Breakthrough: Translation using Lyapunov function

7. Extension to MANETs? How to cope with infinite complexity in Interference models? Issues Complexity from fading – interpreted as infinite resources in a wireless multiple access setting Relaxation in previous work relied on a dominant face in the capacity region. For MANET models this region is smooth TDMA – complexity is nearly infinite for multi-hop interference models. Resulting capacity region again appears smooth

8. Complexity Results in Simplicity Conclusion: Half space relaxation is more easily justified in MANETs D&C Approach Step 1: First identify or approximate rate region near desired operating point. This is the basis of the dynamic fluid model Step 2: Relaxation is again justified through separation of time scales Step 3: Policy synthesis and translation as in 2008 result Step 4: Expand capacity region at hot spots through network coding

9. Challenges CAN WE LEARN? Critical information for optimization is easy to identify. How can this information be shared? CAN WE CODE? With the identification of dynamic bottlenecks, it is then evident where the capacity region can be improved. Summaries and challenges KEY CONCLUSION Complexity in MANETs actually results in a simple model description References S. Meyn. Stability and asymptotic optimality of generalized MaxWeight policies. SIAM J. Con Optim., 47(6):3259– 3294, 2009 W. Chen et. al. Approximate Dynamic Programming using Fluid and Diffusion Approximations with Applications to Power Management. Submitted to the 48th IEEE Conference on Decision and Control, 2009. W. Chen, S. P. Meyn and M. Medard. Optimal Control of Stochastic Networks. Plenary Lecture at Erlang Centennial, April 2009. Manuscript in preparation. S. P. Meyn. Control Techniques for Complex Networks. Cambridge University Press, 2007. F. S. Melo, S. Meyn, and M. I. Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of ICML, pages 664–671, 2008.