Prioritizing Local Inter-Domain Communication in Xen Sisu Xi, Chong Li, Chenyang Lu, and Christopher Gill Cyber-Physical Systems Laboratory Washington University in St. Louis IEEE/ACM International Symposium on Quality of Service, 2013

 Multiple computing elements  Cost! Weight! Power!  Communicate via dedicated network or real-time networks  Use fewer computing platforms to integrate independently developed systems via virtualization Motivation 2

 Multiple computing elements  Cost! Weight! Power!  Communicate via dedicated network or real-time networks  Use fewer computing platforms to integrate independently developed systems via virtualization Motivation 3 Physically Isolated Hosts -> Common Computing Platforms Physically Isolated Hosts -> Common Computing Platforms Network Communication -> Local Inter-Domain Communication Network Communication -> Local Inter-Domain Communication Guarantee QoS with Virtualization???

System Model and Contributions  We focus on  Xen as the underlying virtualization software  Single core for each virtual machine on a multi-core platform  Local Inter-Domain Communication (IDC)  No modification to the guest domain besides the Xen patch  Contributions  Real-Time Communication Architecture (RTCA) in Xen  Reduces high priority IDC latency from ms to us in the presence of low priority IDC 4

B Background – Xen Overview 5 A NIC VMM Scheduler Core VCPU netfront Domain 1 VCPU NIC driver softnet_dat a netback Domain 0 VCPU netfront Domain 2 ……

Part I – VMM Scheduler: Limitations  Default credit scheduler  Schedule VCPUs in round-robin order  RT-Xen scheduling framework  Schedule VCPUs by priority  Server based mechanism, each VCPU has (budget, period)  However  If execution time < 0.5 ms, VCPU budget is not consumed  Solution  Dual quanta: ms for scheduling, while us for time accounting 6 “Realizing Compositional Scheduling through Virtualization”, Real-Time and Embedded Technology and Application Symposium (RTAS), 2012 “Realizing Compositional Scheduling through Virtualization”, Real-Time and Embedded Technology and Application Symposium (RTAS), 2012 “RT-Xen: Towards Real-Time Hypervisor Scheduling in Xen”, ACM International Conferences on Embedded Software (EMSOFT), 2011 “RT-Xen: Towards Real-Time Hypervisor Scheduling in Xen”, ACM International Conferences on Embedded Software (EMSOFT), 2011

Part I – VMM Scheduler: Evaluation 7 VMM Scheduler: RT-Xen VS. Credit C 5 C 0C 1 C 3 C 4 Dom 3Dom 4Dom 0Dom 1Dom 2 Linux % CPU sent pkt every 10ms 5,000 data points C 2 Dom 9Dom 10 … When Domain 0 is not busy, the VMM scheduler dominates the IDC performance for higher priority domains

Part I – VMM Scheduler: Enough??? 8 VMM Scheduler C 5 C 0C 1C 2 C 4 C 3 Dom 3 Dom 4 Dom 5Dom 0Dom 1Dom 2 100% CPU … … …

Part II – Domain 0: Background 9 C D A netfront Domain 1 Domain 0 … B netfront Domain 2 … netif TXRX netback netback[0] { rx_action(); tx_action(); } netfront Domain m … netfront Domain n … … … netif softnet_dat a Packets are fetched in a round-robin order Sharing one queue in softnet_data

Part II – Domain 0: RTCA 10 Packets are fetched by priority, up to batch size A netfront Domain 1 Domain 0 … A netfront Domain 2 … netif TXRX netback netback[0] { rx_action(); tx_action(); } … … softnet_dat a B netfront Domain m … netif netfront Domain n … … netif B Queues are separated by priority in softnet_data

Part II – Domain 0: Evaluation Setup 11 VMM Scheduler C 5 C 0C 1C 2 C 4 C 3 Dom 0Dom 1Dom 2 100% CPU Original vs. RTCA Dom 3Dom 4Dom 5 Interference Medium Heavy Light Base … … … sent pkt every 10ms 5,000 data points

Part II – Domain 0: Latency 12 When there is no interference, IDC performance is comparable Original Domain 0 performs poorly in all cases Due to priority inversion within Domain 0 RTCA with batch size 1 performs best We eliminate most of the priority inversions RTCA with larger batch sizes perform worse under IDC interference IDC Latency between Domain 1 and Domain 2 in presence of low priority IDC (us)

Part II – Domain 0: Latency 13 When there is no interference, IDC performance is comparable Original Domain 0 performs poorly in all cases Due to priority inversion within Domain 0 RTCA with batch size 1 performs best we eliminate most of the priority inversions RTCA with larger bath sizes perform worse under IDC interference By reducing priority inversion in Domain 0, RTCA can effectively mitigate impacts of low priority IDC on the latency of high priority IDC IDC Latency between Domain 1 and Domain 2 in presence of low priority IDC (us)

Part II – Domain 0: Throughput 14 A small batch size leads to significant reduction in high priority IDC latency and improved IDC throughput under interfering traffic iPerf Throughput between Dom 1 and Dom 2

Other Approaches and Future Work  Shared Memory Approach [XWAY, XenLoop, Xensocket]  Required modification to guest OS or applications  Traffic Control in Linux [  Applied within one device. Cannot directly be applied on IDC  Future Work  Multi-Core VM scheduling  Network Interface Card (NIC)  Rate control  Co-ordinate with VMM scheduler 15

Conclusion 16 Hardware VMM Scheduler VCPU netfront Domain 1 VCPU softnet_dat a netback Domain 0 VCPU netfront Domain 2  VMM scheduler alone cannot guarantee IDC latency  RTCA: Real-Time Communication Architecture  RTCA + RT-Xen reduces high priority IDC latency from ms to us in the presence of low priority IDC 

Backup Slides 17

Why IDC? Why Xen? 18  Embedded Systems  Integrated Modular Avionics ARINC 653 Standard Honeywell claims that IMA design can save 350 pounds of weight on a narrow-body jet: equivalent to two adults embedded.org/docs/Events/2007/IMA/Slides/ARTIST2_IMA_WindRiv er_Wilson.pdf Full Virtualization based ARINC 653 partition Sanghyun Han, Digital Avionics Systems Conference (DASC), 2011 Full Virtualization based ARINC 653 partition Sanghyun Han, Digital Avionics Systems Conference (DASC), 2011 ARINC 653 Hypervisor VanderLeest S.H., Digital Avionics Systems Conference (DASC), 2010 ARINC 653 Hypervisor VanderLeest S.H., Digital Avionics Systems Conference (DASC), 2010

Latency Matters to Services  Amazon: Revenue decreased by 1% of sales for every 100 ms latency  costs-you-sales-how-to-crush-it.html costs-you-sales-how-to-crush-it.html  Google: slowing down the search results page by 100 ms to 400 ms has a measurable impact on the number of searches per user of -0.2% to - 0.6%   Firefox: 2.2 seconds faster web response increases 15.4% more Firefox install package download. (equals million additional downloads per year)  19

End-to-End Task Performance 20 VMM Scheduler: Credit vs. RT-Xen C 5 C 0C 1C 2 C 4 C 3 Dom 11Dom 12Dom 13 Interference Medium Heavy Light Dom 0Dom 1Dom 2 100% CPU Original vs. RTCA T1(10, 2) T2(20, 2) T1(10, 2) T3(20, 2) T1(10, 2) T4(30, 2) Dom 1 & Dom 2 60% CPU each Dom 3 to Dom 10 10% CPU each 4 pairs bouncing packets Dom 3 Dom 4 Dom 5 Dom 6 Dom 7 Dom 8 Dom 9 Dom 10 Base

End-to-End Task Performance 21 By combining the RT-Xen VMM scheduler and the RTCA Domain 0 kernel, we can deliver end-to-end real- time performance to tasks involving both computation and communication

Backup – Baseline 22

Domain-0 Domain-U (1). XEN Virtual Network 23 socket(AF_INET, SOCKET_DGRAM, 0); socket(AF_INET, SOCKET_STREAM, 0); sendto(…) recvfrom(…) VMM app kernel TCP IP Netback Driver UDP INET TCP IP Netfront Driver UDP INET Transparent Isolation General Migration XPerformance XData Integrity XMulticast

Domain-U (2). XWay, VEE’08 24 VMM XWAY switch TCP IP XWAY protocol NetfrontXWAY driver UDP INET app kernel socket(AF_INET, SOCKET_DGRAM, 0); socket(AF_INET, SOCKET_STREAM, 0); sendto(…) recvfrom(…) Transparent ? Performance Dynamic Create/Destroy Live Migration XConnect Overhead XPatch Guest OS XNo UDP XComplicated

Domain-U (3). XenSocket, Middleware’07 (IBM) 25 VMM app kernel socket(AF_INET, SOCKET_DGRAM, 0); socket(AF_INET, SOCKET_STREAM, 0); socket(AF_XEN, …); sendto(…) recvfrom(…) TCP IP Netfront UDP INET AF_Xen Netfront No Modification to OS/Xen One way Communication Performance XTransparent

Domain-U (4). XenLoop, HPDC’08 (Binghamton) 26 VMM app kernel socket(AF_INET, SOCKET_DGRAM, 0); socket(AF_INET, SOCKET_STREAM, 0); sendto(…) recvfrom(…) TCP IP Netfront UDP INET XenLoop No Modification to OS/Xen Transparent Performance Migration XOverhead XIsolation ? XDynamic teardown ?

A netfront Domain-1 Domain-0 … A netfront Domain-2 … netif … … softnet_data NIC driver multiple kthreads B netfront Domain-m … netif netfront Domain-n … … netif B TXRX netback TXRX netback TXRX netback priority kthreads highest priority