Quality-based Adaptive Resource Management Architecture (QARMA): A CORBA Resource Management Service Presenter: David Fleeman { D. Fleeman, M. Gillen, A. Lenharth, M. Delaney, L. Welch, D. Juedes and C. Liu Center for Intelligent, Distributed & Dependable Systems Ohio University Athens, OH WPDRTS 2004 April 26, 2004

2 Presentation Overview QARMA –Overview of Architecture –GME/VEST-based Modeling Tool –System Repository Service –Resource Management Service –Enactor Service –Experimental Results Integration Efforts –RT CORBA Dynamic Scheduling Service (URI) –Utah CPU Broker (Utah) Technology Transition –DARPA PCES BBN OEP (UAV) –DARPA ARMS MLRM component –Place services into TAO Conclusions and Future Work

3 Generic Resource Management Architecture Configuration Files Monitors Detectors Information Repository Decision-Maker Enactors Computing Environment & User Applications Environment Resource Instrumentation Software Instrumentation Resource and Software Management Commands

4 CORBA Resource Management Architecture Resource Management Service Enactor Service System Repository Service Configuration Files Specification Tool Resource Management Architecture Application Layer (independent of QoS mechanism) Replaceable Components Enactor (Quality Connector) Host and Network Monitor / Detector Software Performance Monitor / Detector Hosts/Networks Enactor (Quality Connector) Enactor (Quality Connector) Software Performance Monitor / Detector Host and Network Monitor / Detector Host and Network Monitor / Detector TrackingATRUAV Video Resource Management Service Goal Each chain of applications achieves a “benefit” based on it’s service attribute settings (e.g., frame rate) and extrinsic attribute settings (e.g., importance). The Resource Management Service changes service attributes in order to optimize total benefit, subject to the constraint that all tasks meet their real- time requirements. QARMA Components

5 QARMA System Repository Service Purpose –Central repository for communicating all specification and state data to management components Benefits –Reduced complexity of other components –Other components become stateless –Need for replication and fault tolerance of other components is decreased –Flexibility in replacing storage mechanism behind System Repository (e.g., memory, XML files, database) –Integration with components developed by other organizations hinges only on agreement of the data stored in the System Repository Drawbacks –Becomes a single point of failure –Not scalable for arbitrarily large systems –These drawbacks can be handled with standard fault-tolerance and replication mechanisms.

6 QARMA Resource Management Service Purpose –Translate specification into algorithmic model. –Make intelligent allocation decisions. Benefits –Ability to perform global feasibility analysis for tasks that require multiple resources. –Ability to perform global optimization for service quality settings. Drawbacks –Not scalable for arbitrarily large systems. Hierarchical decision-makers, such as the one presented this morning may be used in place of this component.

7 QARMA Resource Management Service Code Snippets IDL Interface module QARMA { … interface RM_Service { typedef sequence Hint; typedef sequence Hint_seq; void reactive_rm(in Hint_seq hints); }; … }; Invocation by Host Detector // Create a hint for the RM Service QARMA::RM_Service::Hint_seq hints; hints.length(1); hints[0].length(2); hints[0][0] = CORBA::string_dup(“Overload”); hints[0][1] = CORBA::string_dup(“host1”); // Invoke the RM Service rm_service->reactive_rm(hints);

8 QARMA Resource Management Service The Key Problems Information Acquisition Information Modeling Resource Allocation Algorithms –Feasibility Testing –Optimization Objective –Control Mechanisms –Pluggable

9 Information Acquisition Obtained from System Repository Service –Resource Specification: Describes hosts, network devices, and network connectivity. –Software Specification: Describes the extrinsic attributes that affect performance of applications and the service attributes that can be used to control application resource usage and quality. Describes the application connectivity and dependencies, performance requirements, and performance characteristics. –Monitoring Data: Host and network resource usage state and statistics Software performance state and statistics

10 Information Modeling Specification data is transformed into mathematical models. –Based on the models presented in chapters 3 and 9 of Jane Liu’s “Real-Time Systems” book. –Service and extrinsic attributes have been added to extend the model presented in the book. Four models are created: –Resource Model –Task Model –Profile Model –Benefit Model

11 Example: Specification => Data Structure => Model Host ou4.cidds { Memory 256 MB; … Network { Interface eth0 { Name “ ”; … } } } Host ou3.cidds { … Network { Interface eth0 { … } } } Switch lan1 { Network { Interface eth0; Interface eth1; } Network Topology { Link {Interface lan1 eth0; Interface ou4.cidds eth0; Speed 100 MB/sec; Latency 0.0 sec; } … } … struct NetworkInterface { string name; … } struct Host { string name; long memory; sequence interfaces; } typedef Host NetworkSwitch; struct NetworkLink { sequence source_interfaces; sequence sink_interfaces; double link_speed; doulbe link_latency; } ou4.cidds with loopback interface ou3.cidds with loopback interface switch lan1 full- duplex link full- duplex link H I L L S L L H I

12 QARMA Control Mechanisms Long term goal is to have all the following: –Service Level Adaptation –Process Allocation (Startup capabilities) –Process Migration (Ability to stop and restart processes) –Process Replication –Dynamic Network Routing –Network Reservation and Prioritization (IntServ / DiffServ) –CPU Reservation and Prioritization (Utah CPU Broker) Currently implemented in QARMA: –Service Level Adaptation –Process Allocation (simplest) –Process Migration (simplest)

13 Example Resource Allocation Algorithm: Greedy Service Level Optimization INPUT: –Resource Model, Task Model, Profile Model, Benefit Model OUTPUT: –Setting of Service Levels such that the resource utilization on any resource is less than 69% and utility is maximized. ALGORITHM –Choose lowest quality settings for all tasks –Rank/order systems by importance –For each system in order of importance: Set service level to the highest While not feasible, decrease service level –Return service level settings

14 QARMA Enactor Service Purpose –Enact changes to the allocation of resources as directed by the Resource Management Service. Benefits –Delegates change directives to lower-level enactors. –Specification for enactors allows enactment mechanisms to be swapped out. Drawbacks –???

15 Integration with the BBN OEP Resource Management Service Enactor Service System Repository Service Configuration Files Specification Tool sys cond QuO Contracts QuO Kernel read Resource Management Architecture sys cond sys cond sys cond Application Layer QuO Middleware QuO Contracts VIDEO DISTRIBUTION HOST Video Distributor Video Display VIDEO DISPLAY HOST Video Display Proxy QuO Video Stream QuO Video Source Process MOBILE VIDEO SOURCE HOST Video File QuO Application Layer BBN OEP UAV System Replaceable Components set Host and Network Monitor / Detector Software Performance Monitor / Detector Event Channel Hosts/Networks ** System Condition objects were added. ** Contracts were modified to allow RM Service to force a chosen region. Host and Network Monitor / Detector Host and Network Monitor / Detector Quality Connector Instance (Quo Enactor)

16 Global Resource Mgmt. Demo Configuration OU1 CORBA Services CORBA RM Services UAV1 Host Mon OU3 Host Mon OU2 Host Mon UAV2 Distrib1 Distrib2 Receiver1 Viewer1 Receiver2 Viewer2 LOAD OU4 There are two Sender-Distributor- Receiver streams. Receiver1 is set to the highest priority (because it is viewing a target). Use Hourglass to apply CPU load on the Receiver node. Load increases every 90 seconds as follows: –40% at time 40 –60% at time 130 –90% at time 220 –98% at time 310 –100% at time 400

Local RMGlobal RM Global RM vs. Local RM - CPU Load Experiment Frame Rate

Frame Rate Average (frames per second) Baseline Viewer15.48 Baseline Viewer26.01 QARMA Viewer18.73 QARMA Viewer22.67 Global RM vs. Local RM - CPU Load Experiment Frame Rate Results

Global RM vs. Local RM - CPU Load Experiment Latency Local RMGlobal RM

Global RM vs. Local RM - CPU Load Experiment Latency Results Latency AverageLatency MaxStandard Deviation Baseline Viewer1236 ms5770 ms624 ms Baseline Viewer2229 ms5751 ms621 ms QARMA Viewer1106 ms3270 ms321 ms QARMA Viewer289 ms3479 ms292 ms Baseline QARMA Average Latency Standard Deviation Baseline V1 Baseline V2 QARMA V1 QARMA V2

Global RM vs. Local RM - CPU Load Experiment Conclusions Applications controlled by the global RM demonstrate greater stability exhibit greater determinism have lower average latency delivered higher frame rate for more important streams higher average frame rate

22 Future Work Integrate other resource management mechanisms Investigate and develop more advanced allocation algorithms Provide fault tolerance and stabilization of the Resource Management Service as well as reflexive management. Continue integration, transition and validation efforts –Validation with RMBench –Validation with PCES OEP (Boeing and BBN) –Integration with PCES technology Scheduling Service, Kokyu, CPU Broker –Transition into DARPA ARMS MLRM infrastructure –Transition into TAO as CORBA Services

23 Envisioned PCES Component Integration... RT CORBA scheduling point Adaptation Parameter Adjustments (i.e., frame rate) Resource Management Service (OU) Enactor Service (OU) System Repository Service (OU) Configuration Files Specification Tool (OU/URI) Resource Management Architecture Other PCES Components Enactor (Quality Connector) Host and Network Monitor / Detector Scheduling Service (URI) Hosts/Networks Host and Network Monitor / Detector Resource Monitoring (KU) Enactor (Quality Connector) Quality Enactor (OU) Application Layer TrackingATRUAV Video RT CORBA Pluggable Scheduler Kokyu (WUSTL) Resource Enactors (KU) Resource Enactor (KU) CPU Broker (Utah)