Cisco Logo


High Performance Computing Networking

One feature of the usNIC ultra-low latency Ethernet solution for the UCS Cisco VIC that we think is interesting is the fact that it is based on SR-IOV.

What is SR-IOV, and why is it relevant in the HPC world?

SR-IOV (Single Root I/O Virtualization) is commonly used in the server virtualization world. The most commonly described purpose of SR-IOV in the hypervisor world is to allow a device partition, called VF (Virtual Function), to be mapped in the guest operating system address space. This allows the guest operating system to enjoy higher I/O performance and lower CPU utilization as compared to the alternative: software-emulated devices that are traditionally implemented in hypervisors.

Compared to the old world before hypervisors came along, that use of SR-IOV seems to allow to regain back some performance lost due to the hypervisor software intervention in the I/O data path. But why should I care about SR-IOV in the world of my network-latency-bound HPC applications running on common operating systems on bare metal servers?

SR-IOV is part of the PCIe standard and it is much more generic than the commonly described use in the hypervisor world suggests:

  1. An SR-IOV compliant device doesn’t require a hypervisor in order to be used. For instance, support for SR-IOV devices and drivers has long been available in the stock Linux kernel.
  2. A VF device doesn’t require the virtual machine monitor software interposition in order to participate in I/O. Instead, the VF is a first class agent on the PCIe bus and, as such, it can originate or claim PCIe transactions autonomously.
  3. A VF device doesn’t require a VM in order to be used. Instead, a VF device can be presented to any address space in the system. For example, the VF device can be presented to a usual unix-style process running in user space.

Ok, but what do these generic provisions of SR-IOV mean for my HPC applications?

They mean that (in reverse order):

Ok, but what about the presence of the VFs on the network?

Well, notwithstanding the common hypervisor case where each VF device has a presence on the network (e.g., its own MAC address), the SR-IOV device model doesn’t require a VF device to have any presence nor identity on the network. A VF device can be anonymous, or invisible, on the network.

One possible arrangement — which we use in the Cisco usNIC solution — is for the PF to have its own presence on the network (e.g., “eth1 with MAC address 11:22:33:44:55:66 and IP address 10.0.0.1″) while the children VFs inherit (or share) the network interface associated to the PF. Note, this scheme doesn’t require bonding/teaming nor any particular intervention of the kernel networking stack.

In summary, SR-IOV contributes to at least three key elements of sustainability in a modern approach to user space networking for HPC applications deployed on common servers, operating systems and networks:

  1. Seamless integration with the IOMMU (which is part of all modern chipsets, x86 and otherwise)
  2. Enabling user space I/O drivers
  3. The non-proliferation of Ethernet or IP network endpoints

Comments Are Closed

  1. Return to Countries/Regions
  2. Return to Home
  1. All High Performance Computing Networking
  2. All Security
  3. Return to Home