We often think about improving application performance by speeding up some part of the application itself. This can be done in many ways, such as optimizing the application code, using a more efficient algorithm, or even offloading parts of the application to specialized hardware. But a lot of the overheads that applications experience today are instead imposed by the underlying hardware interface. In this blog post, we will look at how the interface exposed by existing network interface cards (NICs) imposes significant overheads on modern applications and how a new NIC interface called Ensō can help eliminate these overheads.

This blog post is based on the OSDI ’23 paper Ensō: A Streaming Interface for NIC-Application Communication . Refer to the paper if you are interested in more technical details.

The Rise of the Packetized NIC Interface

Thirty years ago, NICs were significantly simpler. Their only job was to move raw packets between the network and the kernel; the kernel handled all protocol processing (e.g., through the TCP stack), so applications dealt only with processed data. The existing NIC interface was designed around this simple model, where the NIC and the kernel exchange individual packets. We refer to this interface as the packetized NIC interface.

Overview of the Packetized NIC Interface

At a high level, the packetized interface places each incoming and outgoing packet in a dedicated packet buffer in host memory. Each packet buffer has a fixed size that is usually set so that it can accommodate the largest packet size allowed by the protocol (the maximum transmission unit, or MTU). This is necessary, as software does not know ahead of time what the next packet size will be.

The following interactive diagram illustrates how software receives packets from the NIC using a packetized NIC interface. To receive a packet, the software first must post empty packet buffers to the NIC (not shown). The NIC will then keep the addresses of the next available buffers in its internal memory so that, when a packet arrives, the NIC can directly copy the packet to the next available buffer in host memory. Then, for each packet, the NIC sends a descriptor to a descriptor ring buffer, informing the software of the packet’s location in memory.

You can play with the following diagram to explore how the packetized NIC interface works. Click on “Receive” to simulate receiving packets from the NIC and on “Consume” to simulate software consuming the packets. Note that each packet is placed in a dedicated buffer and that the NIC sends a descriptor for every packet received.

The Fall of the Packetized NIC Interface

Many things have changed since the packetized interface was first introduced. On the NIC side, modern NICs have thousands of queues. Applications can now have dedicated queues and rely on the NIC to deliver incoming packets directly to the application’s memory. Modern NICs can also take over tasks that were traditionally done in software (offloads), from simple ones such as checksum computation and segmentation to more complex ones such as full transport protocol implementations. On the software side, applications have also evolved. High-performance network stacks often employ techniques such as batching —where they process multiple packets at a time, instead of individual packets—which helps reduce per-packet overheads.

While NICs and packet processing software have changed dramatically over the last few decades, the interface that NICs expose has remained surprisingly unchanged—still designed to exchange individual packets with software.

The mismatch between the NIC interface and the data being exchanged leads to the following three problems:

❶ Packetized Abstraction: The first problem with the packetized interface is the packetized abstraction itself. This arises from the current trend of NICs increasingly implementing functionality that operates at higher layers of the network stack. NICs that implement a transport protocol are able to push application-level messages or bytestreams—assembled at the NIC by combining multiple packets—directly to software. Unfortunately, by shoehorning these high-level data types into the packetized abstraction, the packetized interface imposes unnecessary overheads on software.

For instance, consider a NIC that implements a transport protocol such as TCP. Implementing transport on the NIC can improve application performance by sparing CPU cycles. With TCP, applications exchange data using bytestreams. TCP is responsible for packetizing the data and making sure that every piece of data sent is received by the application. Therefore, a NIC that implements TCP, or other bytestream-based transport, should be able to reassemble packets and directly push bytestreams to software. But if the NIC exposes a packetized interface, it must split the incoming bytestream into chunks that fit in the available packet buffers. The following interactive diagram illustrates this issue.

Upon receiving these separate chunks, software must recombine them to be able to deliver a contiguous bytestream to the application. This is problematic for two reasons: First, recombining these pieces into a contiguous buffer requires data copies, which consumes CPU cycles. Second, because these pieces can be in arbitrary memory locations, it is hard for the CPU to predict what the next memory access will be. We explore this second problem in more detail next.

❷ Chaotic Memory Accesses: Because the packetized interface places incoming data in packet buffers that can be in arbitrary memory locations, it is hard for the CPU to predict what the next access will be. This prevents CPU features such as the streaming prefetcher—which speculatively loads sequential memory—from working well, leading to a significant number of cache misses. To illustrate this, consider the following interactive diagram that simulates receiving 64 B packets using the packetized interface. Because addresses are unpredictable, whenever software accesses a new packet, it must fetch it from the last-level cache (LLC) or main memory, paying a much higher cost compared to serving data from the L1 cache.

Note that simply arranging the packet buffers sequentially in memory does not solve the problem. This is because packet buffers are fixed-size while packets are not: smaller packets leave holes at the end of their buffers, producing a variable stride between consecutive packets that the CPU cannot predict. Chaotic memory accesses result in as much as a 55% miss ratio for the L2 cache.

❸ Per-Packet Overhead: The packetized interface also adds significant overhead due to per-packet metadata. NICs communicate with the CPU through a PCIe interconnect, whose limited bandwidth must be shared between the data itself and any metadata. Since software needs to post a buffer to the NIC for every packet, and the NIC must send software a descriptor for every packet, the packetized interface consumes a significant fraction of PCIe bandwidth just to exchange metadata. High-performance network stacks such as DPDK, a library for kernel-bypass networking, employ batching. But while this saves CPU cycles, it does nothing to reduce per-packet metadata. As a result, when processing small packets, the bottleneck becomes PCIe rather than the CPU: the system cannot reach the full speed of the network link no matter how many cores it uses.

The following interactive diagram illustrates the issue. Note the PCIe efficiency counter in the bottom right corner, which shows the percentage of PCIe bandwidth used for payload vs. metadata. With small packets (64 B), the PCIe efficiency can be as low as 61%, meaning that 39% of the PCIe bandwidth is used for metadata.

Ensō: A Streaming NIC Interface

Ensō is a new NIC interface that provides a streaming abstraction . At a high level, Ensō allows the software and the NIC to exchange data using bytestreams. Instead of fixed-size buffers, Ensō gives software the illusion of an unbounded buffer through a new primitive called Ensō Pipe. Software can then use Ensō Pipes to exchange data with the NIC, by reading sequentially to receive data, and by writing sequentially to transmit data.

Ensō imposes no structure on the data written to these buffers—applications and the NIC can use Ensō Pipes to communicate arbitrary streams of bytes. This allows Ensō Pipes to be flexibly used regardless of the functionality running on the NIC. NICs that implement no offloads can use Ensō Pipes to communicate raw packets. NICs that implement a message-based transport protocol can push complete messages to these buffers. Finally, NICs that implement a bytestream-based transport protocol, such as TCP, can use Ensō Pipes to communicate bytestreams directly with applications.

How can we implement a streaming abstraction?

Unlike the packetized interface, which uses a ring buffer of descriptors , Ensō uses a ring buffer of data for each Ensō Pipe. Because data is written to the ring buffer itself, applications and the NIC can read and write sequentially.

To synchronize access to the buffer, the NIC and the application each control a pointer. When the application is receiving data from the NIC, the NIC advances its pointer (tail) after writing data to the buffer, and the application advances its pointer (head) when it is done processing the data.

To allow the NIC to send pointer updates to software, Ensō also has a notification buffer. Similar to the descriptor ring buffer in the packetized interface, the notification buffer is a ring buffer with fixed slots. Whenever the NIC wishes to advance its pointer, it sends a notification to software through the notification buffer. Unlike descriptors, notifications do not need to be sent for every chunk of data written to the buffer. Instead, the NIC can send one notification to inform software of multiple chunks of data at once.

The following interactive diagram illustrates the Ensō interface. Note that the packets are written sequentially in the same Ensō Pipe. Also note that the NIC is able to send a single notification for a batch of packets. When the first packet arrives, the NIC sends a notification; it then withholds further notifications until software consumes the first one.

How can a streaming abstraction improve performance?

Besides being a more flexible abstraction for high-level offloads running on the NIC, Ensō’s streaming abstraction also solves the performance issues with the packetized interface that we described earlier.

Streaming abstraction: Bytestreams and large messages no longer need to be split and recombined, eliminating the copies imposed by the packetized interface.

Sequential memory accesses: Since multiple chunks of data are placed back to back in Ensō Pipes, memory accesses are naturally sequential. This makes it easier for the CPU to predict what the next memory access will be. As a result, Ensō vastly reduces the number of cache misses compared to the packetized interface.

No per-packet overhead: Placing data sequentially in Ensō Pipes also allows the NIC to notify multiple chunks of data at once. This avoids the per-packet notification required in the packetized interface. As a result, Ensō significantly reduces the amount of PCIe bandwidth used for metadata as well as CPU cycles required to produce, access, and consume descriptors.

The following interactive diagrams illustrate how Ensō solves the problems we described earlier for the packetized interface.

Implementation

Because Ensō is a new NIC interface, implementing it requires changes to both the NIC and the software running on the CPU. Ensō’s implementation comprises three components:

NIC hardware: We implemented a NIC in SystemVerilog that exposes the Ensō interface. We synthesized the design targeting an FPGA (a programmable hardware device). Using an FPGA lets us test the implementation in a real system, but the design can also be synthesized as fixed-function hardware.

User-space library: The software implementation is designed so that applications can communicate directly with the NIC without going through the kernel. Applications can link to the Ensō library and use its streaming API to push data to or pull data from the NIC.

Kernel module: While applications use the library to communicate directly with the NIC, setup and resource management are still done by the kernel. Ensō provides a kernel module to accomplish these tasks. The library talks with the kernel module whenever it needs to allocate and free resources, e.g., Ensō Pipes.

Impact on Application Performance

Ensō’s performance improvements translate into benefits for real applications. To show this, we ported four different applications to use Ensō and compared their performance with DPDK implementations running with an Intel E810 100 Gb NIC.

The following table summarizes the results. It shows the throughput improvement of the Ensō implementation compared to the original DPDK implementation running with the E810 NIC. We see that Ensō is able to improve throughput by up to 6⨉.

Application Throughput Improvement
Google’s Maglev Load Balancer [ NSDI ’16 ] Up to 6⨉
Network Telemetry with NitroSketch [ SIGCOMM ’19 ] Up to 3.5⨉
MICA Key-Value Store [ NSDI ’14 ] Up to 47%
Log Monitor (Inspired by AWS CloudWatch Logs ) Up to 95%

The results above cover different classes of applications. Google’s Maglev Load Balancer and the network telemetry application are typical applications that operate on raw packets. MICA is a key-value store that operates on messages. Finally, the Log Monitor application is a streaming application that operates on bytestreams. These improvements stem only from the change in NIC interface; we expect even more benefits as Ensō enables NICs to implement more complex offloads such as transport protocols with less software overhead.

The paper includes more detailed experiments: we conduct a series of microbenchmarks that evaluate how some of our design choices affect performance. We also show that Ensō is able to achieve 100 Gbps line rate (the full speed of the link) with minimum-size packets using a single CPU core.

* * *

The packetized interface no longer suits the demands of high-performance applications or the features offered by modern NICs. It is time for NICs to move to a streaming interface. If you would like to use Ensō with your own applications, you can check out its open-source code and documentation . And for more technical details, take a look at the OSDI ’23 paper .