Design and Implementation of Adaptive Signal Processing Systems Using Markov Decision Processes Marilyn Wolf1 With contributions from: Lin Li2, Adrian E. Sapio2, Jiahao Wu2, Yanzhou Liu2, Kyunghun Lee2, and Shuvra S. Bhattacharyya2,3 1. Georgia Institute of Technology, USA 2. University of Maryland, USA 3. Tampere University of Technology, Finland

Outline Motivation and Contribution Background Framework for Design and Implementation of Adaptive Signal Processing Systems Case Study of Channelizer/receiver Application Conclusions

Motivation and Contribution Modern signal processing applications impose increasing demands of: Adaptivity Efficiency Reconfigurability Flexibility Challenges at many levels of system design, implementation and optimization: Dynamically-changing working scenarios Stringent constraints on energy-efficiency and real-time performance Multidimensional design space

Motivation and Contribution This research aims at developing a framework for design and implementation of adaptive embedded signal processing systems that integrates automated, MDP-based generation of optimal reconfiguration policies dataflow-based application modeling implementation of embedded control software that carries out the generated reconfiguration policies

Background Markov Decision Processes (MDPs) Probabilistic transitions combined with inputs. Given an input at a state, next state is chosen probabilistically. A policy p defines the actions in each state s. Optimal policy maximizes rewards.

Background Dataflow Modeling Basic concepts of dataflow modeling: Digital Signal Processing (DSP) system directed dataflow graph Computational functions nodes (actors) Communication channels between actors edges (FIFOs) Actor Firing: Actor execution as a discrete unit of computation Token: The encapsulation of some well-defined amount of data. Consumption/Production Rate: Number of tokens consumed/produced from/to the input/output FIFO during one actor firing.

Background LightWeight DataFlow (LWDF) LightWeight DataFlow (LWDF): a programming methodology for integration, experimentation, and optimization with dataflow modeling approaches. Actor Mode: Determines the dataflow behavior of the actor. Enable function: Checks actor firing condition according to its current mode. This function can be bypassed at run time if static scheduling analysis can ensure the result. Invoke function: Executes an actor firing according to its current mode. Lightweight Dataflow Environment (LIDE): Provides a compact set of application programming interfaces (APIs) that is used for constructing, connecting, and executing dataflow components such as actors, edges, and graphs. LIDE APIs have been implemented in a variety of implementation languages, including C, Verilog, and CUDA

Framework for Adaptive Signal Processing Systems We propose a novel framework, called Hierarchical MDP framework for Compact System-level Modeling (HMCSM).

Framework for Adaptive Signal Processing Systems Hierarchical MDP Subsystem A single Markov decision process (MDP) is transformed into a hierarchy of multiple MDPs that can be independently solved. Such decomposition into a collection of simpler MDPs leads to more efficient design optimization. The stochastic models of the environment and system include, for each of the environment and system, the definition of the state space and the state transition matrix (STM). Control Actions represent the set of possible reconfiguration operations. The Reward Function maps state-action pairs into scores that assess the utility of performing the associated control action during the given state.

Framework for Adaptive Signal Processing Systems Parameterized LIDE Implementation Dataflow graph implementation of the application developed using LIDE Parameter updates are made by setting (at design time or run time) appropriate variables in this implementation Configuration Control Machine (CCM) Determines, based on the current environmental state and system state, whether or not to perform a dynamic reconfiguration operation Determines the specific reconfiguration operation that is to be applied to the system if reconfiguration is to be performed Policy Mapping Engine (MPE) Translates control actions into updates to dynamic parameters in the embedded software

Case Study of Channelizer/receiver Application (1)

Case Study of Channelizer/receiver Application (2) Init Graph: modeling construct in parameterized synchronous dataflow (PSDF) for reconfiguration functionality (determination and propagation of new parameter values). Body Graph: modeling construct in PSDF for core signal processing functionality associated with an application. PSDF specification of channelizer subsystem.

Case Study of Channelizer/receiver Application (3) Average processing power of all the available configurations. Platform for Channelizer: Raspberry Pi 3 Model B Device for Power Measurement: Tektronix Keithley Series 2280 Precision Measurement DC Power Supply

Case Study of Channelizer/receiver Application (4) Comparison among MDP-generated policies and fixed-configuration designs. Simulation results for MDP-I. In addition to providing better energy efficiency compared to the fixed configuration designs, our MDP approach can be configured systematically to generate a much larger set of trade-off options (Pareto-optimized fronts) ensures optimality (with respect to the given reward function)

Solver running time Model size MDP Solver: MATLAB-based open source solver, named MDPSOLVE Platform for MDP Solver: Processor: i7-4710HQ running at 2.50GHz RAM: 12.0 GB OS: 64-bit Windows 10 MDP-I solver running time: 294ms MDP-II-a solver running time: 50.8ms MDP-II-b solver running time: 41.5ms In a deployment with a fixed processing system (MDP-II-b) and changing external environment (MDP-II-a), the hierarchical MDP scheme reduces the solver time from 294ms to 50.8ms, which is a factor of over 5.7X smaller. Model size MDP-I: 1.63MB MDP-II: 265kB MDP-II reduces model size by a factor of over 6.1X.

Conclusions We propose the HMCSM framework for design and implementation of adaptive embedded signal processing systems. HMCSM stands for Hierarchical MDP framework for Compact System-level Modeling. HMCSM provides a structured methodology that integrates dataflow methods; MDP formulation using compact, hierarchical models; optimal policy generation at design time; and dynamic, system-level reconfiguration at run time. We demonstrate the effectiveness of our new MDP-based system design framework through experiments with an adaptive wireless communications receiver.

Thanks!