Asynchronous Control for Coupled Markov Decision Systems Michael J. Neely University of Southern California Information Theory Workshop (ITW) Lausanne,

Slides:

Advertisements

Similar presentations

Michael Alves, Patrick Dugan, Robert Daniels, Carlos Vicuna

Advertisements

1 University of Southern California Keep the Adversary Guessing: Agent Security by Policy Randomization Praveen Paruchuri University of Southern California.

Network Utility Maximization over Partially Observable Markov Channels 1 1 Channel State 1 = ? Channel State 2 = ? Channel State 3 = ? Restless.

Stochastic optimization for power-aware distributed scheduling Michael J. Neely University of Southern California t ω(t)

Delay Reduction via Lagrange Multipliers in Stochastic Network Optimization Longbo Huang Michael J. Neely WiOpt *Sponsored in part by NSF.

Stochastic Network Optimization with Non-Convex Utilities and Costs Michael J. Neely University of Southern California

Intelligent Packet Dropping for Optimal Energy-Delay Tradeoffs for Wireless Michael J. Neely University of Southern California

Dynamic Product Assembly and Inventory Control for Maximum Profit Michael J. Neely, Longbo Huang (University of Southern California) Proc. IEEE Conf. on.

An Introduction to Markov Decision Processes Sarah Hickmott

Distributed Association Control in Shared Wireless Networks Krishna C. Garikipati and Kang G. Shin University of Michigan-Ann Arbor.

Dynamic Index Coding Broadcast Station N N Michael J. Neely, Arash Saber Tehrani, Zhen Zhang University of Southern California Paper available.

Universal Scheduling for Networks with Arbitrary Traffic, Channels, and Mobility Michael J. Neely, University of Southern California Proc. IEEE Conf. on.

Efficient Algorithms for Renewable Energy Allocation to Delay Tolerant Consumers Michael J. Neely, Arash Saber Tehrani, Alexandros G. Dimakis University.

Utility Optimization for Dynamic Peer-to-Peer Networks with Tit-for-Tat Constraints Michael J. Neely, Leana Golubchik University of Southern California.

Stock Market Trading Via Stochastic Network Optimization Michael J. Neely (University of Southern California) Proc. IEEE Conf. on Decision and Control.

Delay-Based Network Utility Maximization Michael J. Neely University of Southern California IEEE INFOCOM, San Diego, March.

Dynamic Optimization and Learning for Renewal Systems Michael J. Neely, University of Southern California Asilomar Conference on Signals, Systems, and.

Dynamic Index Coding User set N Packet set P Broadcast Station N N p p p Michael J. Neely, Arash Saber Tehrani, Zhen Zhang University.

Dynamic Optimization and Learning for Renewal Systems -- With applications to Wireless Networks and Peer-to-Peer Networks Michael J. Neely, University.

Max Weight Learning Algorithms with Application to Scheduling in Unknown Environments Michael J. Neely University of Southern California

Dynamic Data Compression for Wireless Transmission over a Fading Channel Michael J. Neely University of Southern California CISS 2008 *Sponsored in part.

Dynamic Tuning of the IEEE Protocol to Achieve a Theoretical Throughput Limit Frederico Calì, Marco Conti, and Enrico Gregori IEEE/ACM TRANSACTIONS.

Problem Statement Given a control system where components, i.e. plant, sensors, controllers, actuators, are connected via a communication network, design.

*Sponsored in part by the DARPA IT-MANET Program, NSF OCE Opportunistic Scheduling with Reliability Guarantees in Cognitive Radio Networks Rahul.

An Efficient Clustering-based Heuristic for Data Gathering and Aggregation in Sensor Networks Wireless Communications and Networking (WCNC 2003). IEEE,

Energy-Efficient Rate Scheduling in Wireless Links A Geometric Approach Yashar Ganjali High Performance Networking Group Stanford University

Multi-Hop Networking with Hard Delay Constraints Michael J. Neely, University of Southern California DARPA IT-MANET Presentation, January 2011 PDF of paper.

Cross Layer Adaptive Control for Wireless Mesh Networks (and a theory of instantaneous capacity regions) Michael J. Neely, Rahul Urgaonkar University of.

Machine Learning CMPT 726 Simon Fraser University

Code and Decoder Design of LDPC Codes for Gbps Systems Jeremy Thorpe Presented to: Microsoft Research

Lecture outline Support vector machines. Support Vector Machines Find a linear hyperplane (decision boundary) that will separate the data.

1 TCOM 501: Networking Theory & Fundamentals Lectures 9 & 10 M/G/1 Queue Prof. Yannis A. Korilis.

Lecture 10: Support Vector Machines

Chapter 5. Operations on Multiple R. V.'s 1 Chapter 5. Operations on Multiple Random Variables 0. Introduction 1. Expected Value of a Function of Random.

Department of Computer Science Undergraduate Events More

Previously Optimization Probability Review Inventory Models Markov Decision Processes.

Optimal Energy and Delay Tradeoffs for Multi-User Wireless Downlinks Michael J. Neely University of Southern California

A Lyapunov Optimization Approach to Repeated Stochastic Games Michael J. Neely University of Southern California Proc.

COGNITIVE RADIO FOR NEXT-GENERATION WIRELESS NETWORKS: AN APPROACH TO OPPORTUNISTIC CHANNEL SELECTION IN IEEE BASED WIRELESS MESH Dusit Niyato,

Resource Allocation for E-healthcare Applications

Markov Decision Processes1 Definitions; Stationary policies; Value improvement algorithm, Policy improvement algorithm, and linear programming for discounted.

Energy Aware Task Mapping Algorithm For Heterogeneous MPSoC Based Architectures Amr M. A. Hussien¹, Ahmed M. Eltawil¹, Rahul Amin 2 and Jim Martin 2 ¹Wireless.

Transmit Diversity with Channel Feedback Krishna K. Mukkavilli, Ashutosh Sabharwal, Michael Orchard and Behnaam Aazhang Department of Electrical and Computer.

Michael J. Neely, University of Southern California CISS, Princeton University, March 2012 Wireless Peer-to-Peer Scheduling.

1 Chapter 6. Classification and Prediction Overview Classification algorithms and methods Decision tree induction Bayesian classification Lazy learning.

Michael J. Neely, University of Southern California CISS, Princeton University, March 2012 Asynchronous Scheduling for.

Department of Computer Science Undergraduate Events More

Energy-Aware Wireless Scheduling with Near Optimal Backlog and Convergence Time Tradeoffs Michael J. Neely University of Southern California INFOCOM 2015,

Super-Fast Delay Tradeoffs for Utility Optimal Scheduling in Wireless Networks Michael J. Neely University of Southern California

ITMANET PI Meeting September 2009 ITMANET Nequ-IT Focus Talk (PI Neely): Reducing Delay in MANETS via Queue Engineering.

CSE4334/5334 DATA MINING CSE4334/5334 Data Mining, Fall 2014 Department of Computer Science and Engineering, University of Texas at Arlington Chengkai.

Order Optimal Delay for Opportunistic Scheduling In Multi-User Wireless Uplinks and Downlinks Michael J. Neely University of Southern California

Content caching and scheduling in wireless networks with elastic and inelastic traffic Group-VI 09CS CS CS30020 Performance Modelling in Computer.

A Perspective on Network Interference and Multiple Access Control Michael J. Neely University of Southern California May 2008 Capacity Region 

Stochastic Optimization for Markov Modulated Networks with Application to Delay Constrained Wireless Scheduling Michael J. Neely University of Southern.

Department of Computer Science Undergraduate Events More

Delay Analysis for Max Weight Opportunistic Scheduling in Wireless Systems Michael J. Neely --- University of Southern California

Smart Sleeping Policies for Wireless Sensor Networks Venu Veeravalli ECE Department & Coordinated Science Lab University of Illinois at Urbana-Champaign.

Energy Optimal Control for Time Varying Wireless Networks Michael J. Neely University of Southern California

DEPARTMENT/SEMESTER ME VII Sem COURSE NAME Operation Research Manav Rachna College of Engg.

Department of Computer Science Undergraduate Events More

Chapter 10 The Support Vector Method For Estimating Indicator Functions Intelligent Information Processing Laboratory, Fudan University.

Online Fractional Programming for Markov Decision Systems

A Network Virtual Machine for Real-Time Coordination Services

Support Vector Machines Introduction to Data Mining, 2nd Edition by

IEEE Student Paper Contest

energy requests a(t) renewable source s(t) non-renewable source x(t)

Jason A. Fuemmeler, and Venugopal V. Veeravalli Edwin Lei

Throughput-Optimal Broadcast in Dynamic Wireless Networks

Utility Optimization with “Super-Fast”

Presentation transcript:

Asynchronous Control for Coupled Markov Decision Systems Michael J. Neely University of Southern California Information Theory Workshop (ITW) Lausanne, Sept Image Processing Camera Mode Receive Transmit t Device Device 2 Device 1 t0t0 t1t1 t2t2 t3t3 t4t4 t5t5 t6t6 t7t7 t8t8 t 10 t9t9 1 1

Example: Network of Smart Devices Each device m has a Processing Chip and a Wireless Communication Chip. State 4 State 2 State 3 State 1 Processing Chip (device m) time Frame 2 energy bits energy Frame 3 Frame 1 Wireless Comms Chip (device m) Queue Arriving bits time channel quality 2 2

Example: Network of Smart Devices 3 3 There are many such devices sharing wireless resources. Can do opportunistic scheduling.

Example: Network of Smart Devices 4 4 Heterogeneous timelines  we must solve a time averaged fractional optimization: (transmit energy) m + (processing energy) m (frame size) m Minimize: Subject to: (bits generated for link i) m (frame size) m ∑ ∑ ∑ ≤ (transmission) i m m m for all links i in {1,…, L}

General Model 5 5 S separate embedded Markov systems. Each system s in {1, …, S} has state space K (s). Each system has its own (variable length) frames. On frame r for system s, observe:  Observe Random Event ω (s) [r].  Observe Current State k (s) [r].  Choose Control Action α (s) [r]. The 3-tuple (k (s) [r], ω (s) [r], α (s) [r]) determines:  Frame size T (s) [r].  Penalty vector (x (s) [r], y 1 (s) [r], …., y L (s) [r]).  Transition Probabilities P ij (s) [r].

Generalized Goal: 6 6 x (s) T (s) Minimize: Subject to: ∑ s ≤ d i for all penalties i in {1,…, L} y i (s) ∑ s T (s) Fractional terms with different denominators. General problems of this type are intractable. This has special structure that admits an optimal solution.

Theorem 1: 7 7 Consider special case with no random event processes ω (s) [r]. Then: 1.The problem can be transformed into a linear program via a nonlinear change of variables. 2.The total complexity is linear in the number of systems S. Translation: Total complexity is essentially the same as having each system solving its own MDP over its own state space. There is no curse of dimensionality as the number of systems S grows large!

Now Treat Random Events 8 8 Example: L channels, each with quality levels L probabilities for the quality vector ω (s) [r] (cannot estimate this huge number of statistics). Even single “standard” MDPs do not have such random event processes ω (s) [r]. Idea: Use Lyapunov Optimization and Virtual Queues to estimate appropriate scalar max-weight functionals. Theorem 2: This is is a computational tool for total optimality with no curse of dimensionality.

Overview of Algorithm 9 9 Z i [r+1] = max [ Z i [r] + ∑θ (s) [r]y i (s) [r] –d i, 0 ] H k (s) [r+1] = H κ (s) [r] + θ (s) [r]1 κ (s) [r] - ∑θ (s) [r]q ik (s) [r] J (s) [r+1] = J (s) [r] + θ (s) [r]T (s) [r] - 1 s i Virtual Queue Update (for system s, penalty i, state k): Use a drift-plus-penalty (or “max-weight”) decision to choose actions on each frame based on virtual queue values and observed random events ω (s) [r]. θ (s) [r] is an auxiliary variable related to 1/(frame size).