© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Achieving 10 Gb/s Using Xen Para-virtualized.

Slides:



Advertisements
Similar presentations
Diagnosing Performance Overheads in the Xen Virtual Machine Environment Aravind Menon Willy Zwaenepoel EPFL, Lausanne Jose Renato Santos Yoshio Turner.
Advertisements

Redesigning Xen Memory Sharing (Grant) Mechanism Kaushik Kumar Ram (Rice University) Jose Renato Santos (HP Labs) Yoshio Turner (HP Labs) Alan L. Cox (Rice.
Virtual Switching Without a Hypervisor for a More Secure Cloud Xin Jin Princeton University Joint work with Eric Keller(UPenn) and Jennifer Rexford(Princeton)
XEN AND THE ART OF VIRTUALIZATION Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, lan Pratt, Andrew Warfield.
Bart Miller. Outline Definition and goals Paravirtualization System Architecture The Virtual Machine Interface Memory Management CPU Device I/O Network,
Performance Evaluation of Open Virtual Routers M.Siraj Rathore
1 Ally: OS-Transparent Packet Inspection Using Sequestered Cores Jen-Cheng Huang 1, Matteo Monchiero 2, Yoshio Turner 3, Hsien-Hsin Lee 1 1 Georgia Tech.
CS 333 Introduction to Operating Systems Class 11 – Virtual Memory (1)
Xen and the Art of Virtualization A paper from the University of Cambridge, presented by Charlie Schluting For CS533 at Portland State University.
Network Implementation for Xen and KVM Class project for E : Network System Design and Implantation 12 Apr 2010 Kangkook Jee (kj2181)
1 Last Class: Introduction Operating system = interface between user & architecture Importance of OS OS history: Change is only constant User-level Applications.
Embedded Transport Acceleration Intel Xeon Processor as a Packet Processing Engine Abhishek Mitra Professor: Dr. Bhuyan.
1  1998 Morgan Kaufmann Publishers Chapter Seven Large and Fast: Exploiting Memory Hierarchy (Part II)
FreeBSD Network Stack Performance Srinivas Krishnan University of North Carolina at Chapel Hill.
1 OS & Computer Architecture Modern OS Functionality (brief review) Architecture Basics Hardware Support for OS Features.
Virtualization for Cloud Computing
Xen and the Art of Virtualization Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew Warfield.
Xen and the Art of Virtualization. Introduction  Challenges to build virtual machines Performance isolation  Scheduling priority  Memory demand  Network.
Sven Ubik, Petr Žejdl CESNET TNC2008, Brugges, 19 May 2008 Passive monitoring of 10 Gb/s lines with PC hardware.
Zen and the Art of Virtualization Paul Barham, et al. University of Cambridge, Microsoft Research Cambridge Published by ACM SOSP’03 Presented by Tina.
Support for Smart NICs Ian Pratt. Outline Xen I/O Overview –Why network I/O is harder than block Smart NIC taxonomy –How Xen can exploit them Enhancing.
SAIGONTECH COPPERATIVE EDUCATION NETWORKING Spring 2009 Seminar #1 VIRTUALIZATION EVERYWHERE.
Virtualization The XEN Approach. Virtualization 2 CS5204 – Operating Systems XEN: paravirtualization References and Sources Paul Barham, et.al., “Xen.
LWIP TCP/IP Stack 김백규.
Benefits: Increased server utilization Reduced IT TCO Improved IT agility.
1 XenSocket: VM-to-VM IPC John Linwood Griffin Jagged Technology virtual machine inter-process communication Suzanne McIntosh, Pankaj Rohatgi, Xiaolan.
Xen I/O Overview. Xen is a popular open-source x86 virtual machine monitor – full-virtualization – para-virtualization para-virtualization as a more efficient.
LWIP TCP/IP Stack 김백규.
Xen I/O Overview.
Improving Network I/O Virtualization for Cloud Computing.
© 2010 IBM Corporation Plugging the Hypervisor Abstraction Leaks Caused by Virtual Networking Alex Landau, David Hadas, Muli Ben-Yehuda IBM Research –
Lecture 19: Virtual Memory
Penn State CSE “Optimizing Network Virtualization in Xen” Aravind Menon, Alan L. Cox, Willy Zwaenepoel Presented by : Arjun R. Nath.
Xen and The Art of Virtualization Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt & Andrew Warfield.
LECC2003 AmsterdamMatthias Müller A RobIn Prototype for a PCI-Bus based Atlas Readout-System B. Gorini, M. Joos, J. Petersen (CERN, Geneva) A. Kugel, R.
Politecnico di Torino Dipartimento di Automatica ed Informatica TORSEC Group Performance of Xen’s Secured Virtual Networks Emanuele Cesena Paolo Carlo.
The Memory Hierarchy 21/05/2009Lecture 32_CA&O_Engr Umbreen Sabir.
CS533 Concepts of Operating Systems Jonathan Walpole.
Nathanael Thompson and John Kelm
Srihari Makineni & Ravi Iyer Communications Technology Lab
High Performance Network Virtualization with SR-IOV By Yaozu Dong et al. Published in HPCA 2010.
Next Generation Operating Systems Zeljko Susnjar, Cisco CTG June 2015.
TCP Offload Through Connection Handoff Hyong-youb Kim and Scott Rixner Rice University April 20, 2006.
Intel Research & Development ETA: Experience with an IA processor as a Packet Processing Engine HP Labs Computer Systems Colloquium August 2003 Greg Regnier.
Introduction to virtualization
Processes and Virtual Memory
Full and Para Virtualization
Virtual Memory Review Goal: give illusion of a large memory Allow many processes to share single memory Strategy Break physical memory up into blocks (pages)
CSE598c - Virtual Machines - Spring Diagnosing Performance Overheads in the Xen Virtual Machine EnvironmentPage 1 CSE 598c Virtual Machines “Diagnosing.
Exploiting Task-level Concurrency in a Programmable Network Interface June 11, 2003 Hyong-youb Kim, Vijay S. Pai, and Scott Rixner Rice Computer Architecture.
© 2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Understanding Virtualization Overhead.
Running Commodity Operating Systems on Scalable Multiprocessors Edouard Bugnion, Scott Devine and Mendel Rosenblum Presentation by Mark Smith.
Memory Virtualization In Xen
Virtualization for Cloud Computing
Xen and the Art of Virtualization
Zero-copy Receive Path in Virtio
Presented by Yoon-Soo Lee
Memory COMPUTER ARCHITECTURE
Xen: The Art of Virtualization
CSE 153 Design of Operating Systems Winter 2018
OS Virtualization.
Lecture 14 Virtual Memory and the Alpha Memory Hierarchy
An NP-Based Router for the Open Network Lab Overview by JST
Xen Network I/O Performance Analysis and Opportunities for Improvement
Morgan Kaufmann Publishers Memory Hierarchy: Virtual Memory
VIRTIO 1.1 FOR HARDWARE Rev2.0
CSE 153 Design of Operating Systems Winter 2019
Xen and the Art of Virtualization
System Virtualization
ARM920T Processor This training module provides an introduction to the ARM920T processor embedded in the AT91RM9200 microcontroller.We’ll identify the.
Presentation transcript:

© 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Achieving 10 Gb/s Using Xen Para-virtualized Network Drivers Kaushik Kumar Ram*, J. Renato Santos +, Yoshio Turner +, Alan L. Cox*, Scott Rixner* + HP Labs *Rice University

Xen summit – Feb /25/ Xen PV Driver on 10 Gig Networks Focus of this talk: RX Throughput on a single TCP connection (netperf)

Xen summit – Feb /25/ Network Packet Reception in Xen Driver DomainGuest Domain Backend Driver Fronten d Driver Xen Physical Driver Hardware NIC I/O Channel Incoming Pkt IRQ Bridge grant copy event DMA demux Push into the network stack 1 Post grant on I/O channel gr Mechanisms to reduce driver domain cost: Use of Multi-queue NIC −Avoid data copy −Packet demultiplex in hardware Grant Reuse Mechanism −Reduce cost of grant operations

Xen summit – Feb /25/ Using Multi-Queue NICs Driver Domain Guest Domain Backend Driver Fronten d Driver Xen Physical Driver Hardware MQ NIC I/O Channels Incoming Pkt IRQ event 6 8 Post grant on I/O channel 1 Map buffer post buf on dev queue DMA 5 UnMap buffer 7 9 Push into the network stack gr 3 2 Advantage of multi-queue Avoid data copy Avoid software bridge One RX queue per guest guest MAC addr demux 4

Xen summit – Feb /25/ Performance Impact of Multi-queue Savings due to multiqueue grant copy bridge Most of remaining cost grant hypercalls (grant + xen functions) Driver Domain CPU Cost

Fronten d Driver Xen summit – Feb /25/ Using Grants with Multi-queue NIC Driver DomainGuest Domain Backend Driver Xen Physical Driver NIC 1 map grant hypercall gr 3 unmap grant hypercall Multi-queue replaces one grant hypercall (copy) with two hypercalls (map/unmap) Grant hypercalls are expensive Map/unmap calls for every I/O operation use page for I/O 2

Xen summit – Feb /25/ Reducing Grant Cost Grant Reuse −Do not revoke grant after I/O is completed −Keep buffer page on a pool of unused I/O pages −Reuse already granted pages available on buffer pool for future I/O operations −Avoids map/unmap on every I/O

Xen summit – Feb /25/ Revoking a Grant for when the Page is Mapped in Driver Domain Guest may need to reclaim I/O page for other use (e.g. memory pressure on guest) Need to unmap page at driver domain before using it in guest kernel To preserve memory isolation (e.g. protect from driver bugs) Need handshake between frontend and backend to revoke grant This may be slow especially if the driver domain is not running

Xen summit – Feb /25/ Approach to Avoid Handshake when Revoking Grants Observation: No need to map guest page into driver domain with multi-queue NIC Software does not need to look at packet header, since demux is performed in the device Just need page address for DMA operation Approach: Replace grant map hypercall with a shared memory interface to the hypervisor Shared memory table provides translation of guest grant to page address No need to unmap page when guest needs to revoke grant (no handshake)

Xen summit – Feb /25/ Software I/O Translation Table Driver DomainGuest Domain Backend Driver Fronten d Driver Xen Physical Driver NIC create a grant for buffer page Send grant over I/O channel 2 set hypercall Validate, pin and update SIOTT 9 clear hypercall 6 get page 8 reset use 1 3 SIOT T #pg use pg 0 1 Use page for I/O 7 DMA event 10 check use and revoke gr 4 5 set use pg SIOTT: software I/O translation table −Indexed by grant reference −“pg” field: guest page address & permission −“use” field indicates if grant is in use by driver domain set/clear hypercalls −Invoked by guest −Set validates grant, pins page, and writes page address to SIOTT −Clear requires that “use”=0

Xen summit – Feb /25/ Grant Reuse: Avoid pin/unpin hypercall on every I/O Driver DomainGuest Domain Backend Driver Fronten d Driver Xen Physical Driver NIC create grant 2 set hypercall validate, pin and update SIOTT 1 3 SIOTT #pg use pg 0 event I/O Buffer Pool reuse buffer & grant from pool 5 return buffer to pool & keep grant 4 kernel mem pressure clear hypercall 8 9 clear SIOT return page to kernel 7 11 gr 6 return buffer to pool & keep grant 10 revoke grant Use page for I/O

Xen summit – Feb /25/ Performance Impact of Grant Reuse w/ Software I/O Translation Table cost saving: grant hypercall Driver Domain CPU Cost

Xen summit – Feb /25/ Impact of optimizations on throughput Data rate CPU utilization Multi-queue w/ grant reuse significantly reduce driver domain cost Bottleneck shifts from driver domain to guest Higher cost in guest than in Linux still limits throughput in Xen

Xen summit – Feb /25/ Additional optimizations at guest frontend driver LRO (Large Receive Offload) support at frontend −Consecutive packets on same connection combined into one large packet −Reduces cost of processing packet in network stack Software prefetch −Prefetch next packet and socket buffer struct into CPU cache while processing current packet −Reduces cache misses at frontend Avoid full page buffers −Use half-page (2KB) buffers (Max pkt size is 1500 bytes) −Reduces TLB working set and thus TLB misses

Xen summit – Feb /25/ Performance impact of guest frontend optimizations Optimizations bring CPU cost in guest close to native Linux Remaining cost difference −Higher cost in netfront than in physical driver −Xen functions to send and deliver events Guest Domain CPU Cost

Xen summit – Feb /25/ Impact of all optimizations on throughput Multiqueue with software optimizations achieves the same throughput as direct I/O ( ~8 Gb/s) 2 or more guests are able to saturate 10 gigabit link current PV driver optimized PV driver (1 guest) optimized PV driver (2 guests) Direct I/O (1 guest) Linux

Xen summit – Feb /25/ Conclusion Use of multi-queue support in modern NICs enables high performance networking with Xen PV Drivers −Attractive alternative to Direct I/O Same throughput, although with some additional CPU cycles at driver domain Avoids hardware dependence in the guests −Light driver domain enables scalability for multiple guests Driver domain can now handle 10 Gb/s data rates Multiple guests can leverage multiple CPU cores and saturate 10 gigabit link

Xen summit – Feb /25/ Status −Performance results obtained on a modified netfront/netback implementation using the original Netchannel1 protocol −Currently porting mechanisms to Netchannel2 Basic multi-queue already available on public netchannel2 tree Additional software optimizations still in discussion with community and should be included in netchannel2 sometime soon. Thanks to −Mitch Williams and John Ronciak from Intel for providing samples of Intel NICs and for adding multi-queue support on their driver −Ian Pratt, Steven Smith and Keir Fraser for helpful discussions