Using Prioritized Network Traffic to Achieve End-to-End Predictability BBN Technologies OOMWorks LLC Cambridge, MA Metuchen, NJ Craig Rodrigues Yamuna Krishnamurthy Irfan Pyarali Pradeep Gore Real-Time and Embedded Distributed Object Computing June 15-18, 2002 Arlington, Virginia, USA

Managed Behavior for Computational Resources In order to preserve end-to-end Quality of Service (QoS) requirements, different technologies must be used to manage the behavior of computational resources Processor Resources –RT-CORBA Network Resources –RSVP –DiffServ

Talk Outline Real-Time CORBA (RTCORBA) Overview RSVP and Diffserv Overviews Diffserv enhancements to RTCORBA middleware Performance results

Real-time CORBA Overview RT-CORBA adds QoS control to regular CORBA to improve the application predictability –Bounding priority inversions –Managing resources end-to-end Policies & mechanisms for resource configuration/control in RT-CORBA include: Processor Resources –Thread pools –Priority models –Portable priorities Communication Resources –Protocol policies –Explicit binding Memory Resources –Request buffering These capabilities address some (but by no means all) important real-time application development challenges

Preserving Priorities End-to-End RT-CORBA preserves end-to-end priorities by: –Mapping importance of activities to corresponding OS priorities –Propagating priorities across the network as activity spans multiple hosts However, RT-CORBA specification is less explicit about: —Communication transport and underlying network Unless this behavior is carefully considered and modeled: — In-transit activities will be affected by network idiosyncrasies — End-to-end predictability in the system difficult to achieve

What is RSVP? Resource Reservation Protocol, specified in IETF RFC 2205 QoS properties are requested by an “out-of-band” signalling protocol Can be used to request a reserved bandwidth between a sender and receiver host Information about the reservation is stored in each intermediate router: “soft-state”

Problems with RSVP Protocol is very complex to implement – much complexity due to anticipated use-cases of large scale multicast trees; multicast not widely used Storing reservation per-flow reservation state is costly – some work being done into looking reservations for aggregate flows Microsoft implemented RSVP in Win2K, disabled it in WinXP – equipment vendors have less incentive to implement it

Problems with RSVP (continued) IETF Next Steps in Signaling (NSIS) Working group – realized that RSVP has it shortcomings, but a signaling protocol for QoS is useful – developing new signaling mechanism (“RSVP Lite”?) – specification due end of 2002 Maybe worth looking at in future, but right now industry seems to be moving away RSVP

Solution: Differentiated Services Use Differentiated Services to prioritize the RT-CORBA network traffic Differentiated Services (Diffserv) architecture provides different types or levels of service for network traffic Diffserv Code Points (DSCP) are added to data packet headers to specify the expected type of service Diffserv enabled routers and network elements use DSCP codepoints to differentiate the network traffic We are enhancing the existing TAO ORB's real-time implementation by making it Diffserv aware

What is Diffserv? Diffserv Field (8-bits) Header Length (4-bits) Total Length (16-bits) Source Address (32-bits) Destination Address (32-bits) TTL (8-bits)Protocol (8-bits)Checksum (16-bits) Flags (3-bits) Version (4-bits) Identification (16-bits)Fragment offset (13-bits) Diffserv field: 6 bits of Diffserv Codepoint, 2 bits ECN Each DSCP (0-63) specifies a Per-Hop-Behavior (PHB), which is a type of router-level QoS (RFC 2475) IP Datagram Header

Diffserv QoS Enhancements to the TAO ORB Provided an efficient and flexible way of setting the DSCP codepoints in the ORB data packets –Done by extending the RTCORBA protocol properties –Setting the DSCP codepoint in the message sent and the reply received Provided a mechanism to map the RT-CORBA priorities to the Diff-Serv network priorities

RTCORBA IDL Modification local interface TCPProtocolProperties : ProtocolProperties { attribute long send_buffer_size; attribute long recv_buffer_size; attribute boolean keep_alive; attribute boolean dont_route; attribute boolean no_delay; attribute boolean enable_network_priority; };

Initialize Protocol Properties //Set the tcp protocol properties RTCORBA::TCPProtocolProperties_var tcp_properties = rt_orb->create_tcp_protocol_properties (send_buffer_size, recv_buffer_size, 1, // keep_alive 0, // dont_route 1, // no_delay enable_network_priority);

Client Side – Initializing the Policy List RTCORBA::ProtocolList protocols; protocols.length (1); protocols[0].protocol_type = 0; protocols[0].transport_protocol_properties = RTCORBA::ProtocolProperties::_duplicate (tcp_properties.in ()); protocols[0].orb_protocol_properties = RTCORBA::ProtocolProperties::_nil (); poa_policy_list[1] = rt_orb->create_client_protocol_policy (protocols );

Client Side - Change Transport Protocol Policy ORB Level object = orb->resolve_initial_references ("ORBPolicyManager"); CORBA::PolicyManager_var policy_manager = CORBA::PolicyManager::_narrow (object.in () ); policy_manager->set_policy_overrides (policy_list, CORBA::SET_OVERRIDE );

Client Side - Change Transport Protocol Policy Thread Level orb->orb_core () >policy_current().set_policy_overrides (policy_list, CORBA::SET_OVERRIDE); Object Level CORBA::Object_var object = server->_set_policy_overrides (policy_list, CORBA::SET_OVERRIDE); server = Test::_narrow (object.in () );

Server Side - Initializing the Policy List RTCORBA::ProtocolList protocols; protocols.length (1); protocols[0].protocol_type = 0; protocols[0].transport_protocol_properties = RTCORBA::ProtocolProperties::_duplicate (tcp_properties.in ()); protocols[0].orb_protocol_properties = RTCORBA::ProtocolProperties::_nil (); poa_policy_list[1] = rt_orb->create_server_protocol_policy (protocols );

Server Side – Set Policy on the POA //Create POA with Diffserv enabled PortableServer::POA_var poa_with_ds = root_poa->create_POA ("POA_WITH_DS", poa_manager.in (), poa_policy_list);

RTCORBA to Network Priority Mapping Added a pluggable RTCORBA to Network Priority Mapping framework Similar to the RTCORBA to Thread Priority Mapping framework RTCORBA IDL modification: typedef short NetworkPriority; native NetworkPriorityMapping; The framework allows the user to implement their own mapping algorithms and plug it in using the ACE Service Configurator

RTCORBA/Diffserv Testbed * * Diffserv T1 T2 Linux Priority of T1 = Priority of T2 IIOP Congestion Traffic

Performance Results 1.No network traffic congestion 2.With network traffic congestion 3.With network traffic congestion and DSCP set on Thread 1

Performance Results With congestion traffic and DSCP set on T1

Potential Use Case for RT-CORBA/Diffserv “Predator” Squadron Operations Center ADSI with JCPI & PowerScene Equipped Air Operations Center Air Defense System Integrator With JCPI STU III Back-Up Nellis – ISAFAF Fiber Optical Link Primary TADIL & Intel Feeds UAV Precise Position Location Information PPLI Prioritized Data Feeds GCS #1GCS #2GCS #3GCS #4 JTIDS Network ISR Constellation UAV PPLI Source: UAV Battlelab, Eglin Air Force Base U.S. Air Force

Summary End-to-end QoS requires resource management of CPU and network resources RT-CORBA today primarily manages CPU resources Our enhancement allows RT-CORBA middleware to manage network resources using Diffserv, which is priority based

Obtaining Software Contacts: –Craig Rodrigues, –Yamuna Krishnamurthy,

Further References “Protocol Selection and Explicit Binding”, Schmidt & Vinoski “An Architectural for Differentiated Services”, RFC