Systems & Networking
building next-generation datacenter network systems

Advance in datacenter networking in the past decade has driven a sea change in the way datacenters are organized and managed. We are exploring various datacenter networking issues from the systems perspective, and we are rethinking datacenter network systems for new hardware and software trends. Some of our current focuses include network for resource disaggregation and RDMA network.

Disaggregating Network Functionalities: A Consolidation Approach with SuperNIC

Prior resource disaggregation works including our own demonstrated how to disaggregate compute, memory, and storage resources. We, for the first time, demonstrate how to disaggregate network resources by proposing a new distributed hardware framework called SuperNIC. Each SuperNIC connects a small set of endpoints and consolidates network functionalities for these endpoints. We prototyped SuperNIC with FPGA and demonstrate its performance and cost benefits with real network functions and customized disaggregated applications.

RDMA Side-Channel Attack

RDMA is a technology that allows direct access from the network to a machine’s main memory without involving its CPU. While RDMA provides massive performance boosts and has thus been adopted by several major cloud providers, security concerns have so far been neglected.

The need for RDMA NICs to bypass CPU and directly access memory result in them storing various metadata like page table entries in their on-board SRAM. When the SRAM is full, RNICs swap metadata to main memory across the PCIe bus. We exploited the resulting timing difference to establish side channels and demonstrated that these side channels can leak access patterns of victim nodes to other nodes.

Pythia is a set of RDMA-based remote sidechannel attacks that allow an attacker on one machine to learn how victims on other machines access the server’s exported in-memory data. We reverse engineered the memory architecture of the most widely used RDMA NIC and use this knowledge to improve the efficiency of Pythia. We further extended Pythia to build side-channel attacks on Crail, a real RDMA-based key-value store application. Pythia is fast (57μs), accurate (97% accuracy), and can hide all its traces from the victim or the server.

Datacenter Approximate Tranmission Protocol

Many datacenter applications such as machine learning and streaming systems do not need the complete set of data to perform their computation. Current approximate applications in datacenters run on a reliable network layer like TCP and either sample data before sending or drop data after receiving to improve performance. These approaches are network oblivious and transmit (and retransmit) more data than necessary, affecting both application runtime and network bandwidth usage.

We propose to run approximate applications on a lossy network and to allow packet loss in a controlled manner. We designed a new network protocol called Approximate Transmission Protocol, or ATP, for datacenter approximate applications. ATP opportunistically exploits available network bandwidth as much as possible, while performing a loss-based rate control algorithm to avoid bandwidth waste and retransmission. It also ensures bandwidth fair sharing across flows and improves accurate applications’ performance by leaving more switch buffer space to accurate flows.

Indirection Layer for RDMA

Recently, there is an increasing interest in building datacenter applications with RDMA because of its low-latency, high-throughput, and low-CPU-utilization benefits. However, RDMA is not readily suitable for datacenter applications. It lacks a flexible, high-level abstraction; its performance does not scale; and it does not provide resource sharing or flexible protection. Because of these issues, it is difficult to build RDMA-based applications and to exploit RDMA’s performance benefits.

To solve these issues, we built LITE, a Local Indirection TiEr for RDMA that virtualizes native RDMA into a flexible, high-level, easy-to-use abstraction and allows applications to safely share resources.

Find out more about and get LITE here.

Related Publication

Conferences and Journals

Disaggregating and Consolidating Network Functionalities with SuperNIC
Yizhou Shan, Will Lin, Ryan Kosta, Arvind Krishnamurthy, Yiying Zhang
arXiv:2109.07744 (arXiv '21)

Exploiting Network Loss for Distributed Approximate Computing with NetApprox
Ke Liu, Jinmou Li, Shin-Yeh Tsai, Theophilus Benson, Yiying Zhang
arXiv:1901.01632 (arXiv '19)

Pythia: Remote Oracles for the Masses
Shin-Yeh Tsai, Mathias Payer, Yiying Zhang
Proceedings of the 28th USENIX Security Symposium (USENIX SEC '19)

LITE Kernel RDMA Support for Datacenter Applications
Shin-Yeh Tsai, Yiying Zhang
Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '17)


Towards a Fully Disaggregated and Programmable Data Center
Yizhou Shan, Will Lin, Zhiyuan Guo, Yiying Zhang
to appear at the 13th ACM Asia-Pacific Workshop on Systems (APSys '22)

User-Defined Cloud
Yiying Zhang, Ardalan Amiri Sani, Guoqing Harry Xu
The 18th Workshop on Hot Topics in Operating Systems (HotOS '21)

A Double-Edged Sword: Security Threats and Opportunities in One-Sided Network Communication
Shin-Yeh Tsai, Yiying Zhang
11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '19)

Building Atomic, Crash-Consistent Data Stores with Disaggregated Persistent Memory
Shin-Yeh Tsai, Yiying Zhang
the 10th Annual Non-Volatile Memories Workshop (NVMW '19)

MemAlbum: an Object-Based Remote Software Transactional Memory System
Shin-Yeh Tsai, Yiying Zhang
the 2018 Workshop on Warehouse-scale Memory Systems (WAMS '18) (co-located with ASPLOS '18)