High Performance VMMBypass IO in Virtual Machines

1
High Performance VMM-Bypass I/O in Virtual
Machines

• Jiuxing Liu, Wei Huang, Bulent Abali,
  Dhabaleshwar Panda
• High Performance Communication (CS8803)
• Paper Presentation
• Adit Ranadive

2
VMs and VMM

• Introduced in 1960s
• Key component Virtual Machine Monitor
  (Hypervisor)
• Allows running different VMs on a single physical
  machine
• Examples VMWare ESX Server, Denali, Xen

3
Xen Virtual Memory Monitor

• Developed by the Systems Research Group of the
  University of Cambridge Computer Lab

4
Xen Split Driver Architecture
5
High Speed Interconnects

• Very low latencies (lt5µs)
• Examples Myrinets, Quadrics, Infiniband, VIA
• Communication protocol offloaded to NIC
• Provide OS bypass I/O
• Low CPU overhead, reduced latency
• Benefits to VMs

6
Proposed Idea

• Based on OS bypass I/O design of interconnects
  (Infiniband)
• Overcome problems of Infiniband in virtualized
  environments
• High latencies
• Higher CPU overhead
• I/O accesses
• VMM is bottleneck

7
Infiniband Architecture

• Host Channel Adapters -gt NICS
• Queue-based model for communication
• Queue Pair (QP), Work Queue Request (WQR),
  Completion Queues (CQ), Completion Queue Entries
  (CQE)
• Notifications using doorbells
• Management Datagrams (MADs)
• Buffer management TPT, Pinning

8
Infiniband Interface
9
VMM Bypass I/O

• Problem of Device virtualization
• Para-virtualizing devices

10
VMM Bypass I/O (contd)

• I/O handled by another VM

11
Prototype Design
Xen-IB driver with split driver model
12
Prototype Design

• Need direct access to HCA from guest VM
• Allowing Infiniband Privileged Accesses
• Initialization and resource management Opening
  HCA, Creating QP, Creating CQ
• Memory registration Memory regions to be used
• Event handling Communication between CQ/QP and
  guest VM

13
VMM-Bypass Accesses

• QP Accesses
• Completion of communication
• CQ Polling
• Both in critical path
• Requirements
• UAR pages available to guest domain
• QP and CQ buffers visible to guest domain

14
VMM-Bypass Accesses

• UAR Mapping implemented by front-end driver
• Loading of front-end driver
• Start of user application
• 2 stage model for QP/CQ buffers
• Registration through IDD
• Frontend sends CQ/QP creation commands to IDD
• Virtualizing Infiniband Management Operation

15
VMM-Bypass Accesses
Design of IB driver
16
Performance Evaluation

• Setup
• Infiniband Cluster
• Dual Intel Xeon 3.0 Ghz CPUs, 2 GB Memory
• Mellanox MT23108 PCI-X IB HCA
• Infiniband Latency and Bandwidth
• Event/Interrupt Handling Overhead
• Memory Registration
• IPoIB Performance
• MPI Performance

17
Infiniband Latency
Infiniband Send/Receive Latency
Infiniband RDMA Write Latency
18
Infiniband Bandwidth
Infiniband RDMA Write Bandwidth
Infiniband Send/Receive Bandwidth
19
Event/Interrupt Handling Overhead
Inter-domain Communication One way Latency
Send/Receive Latency using Blocking VERBS API
20
Memory Registration and IPoIB Performance
Memory Registration Time
IPoIB Netperf Throughput
21
MPI Performance
MPI Latency
MPI Bandwidth
MPI NAS Benchmark
22
Limitations

• Safe Device Access
• All privileged accesses through Backend
• UAR pages contain virtual access points
• Limit size of buffers and allow frequent reuse
• Performance Isolation
• QoS among different VMs
• VM Check-pointing Migration
• Insufficient knowledge in VMM
• VM state stored on I/O device