Publications of Torsten Hoefler

Jiayong Li, Jonas Dann, Zhenhao He, Gustavo Alonso, Sai Rahul Chalamalasetti, Dejan Milojicic, Lance Evans, Alex Veprinsky, Runbin Shi:

StreamDedup: Distributed In-line Deduplication for Disaggregated Storage

(ACM Trans. Reconfigurable Technol. Syst.. presented in New York, NY, USA, ACM, ISSN: 1936-7406, Mar. 2026, Just Accepted )

Publisher Reference

Abstract

Efficient data reduction techniques, including deduplication and compression, are essential in storage systems, affecting performance and longevity. Existing data deduplication approaches often focus on intra-SSD deduplication, missing opportunities for cross-node deduplication, or have scalability issues when aiming for low latency and high throughput data reduction on large-scale, distributed SSD arrays. We propose StreamDedup, a distributed stream accelerator implementing a transparent layer of deduplication as a network-attached, middle-tier service between the compute and storage tiers. StreamDedup manages all aspects of data deduplication and compression and can be seamlessly integrated into existing systems. It is RDMA-enabled and highly scalable, enhancing data processing capacities for large-scale storage systems. Our prototype, deployed on FPGAs, demonstrates that StreamDedup achieves a throughput of 12.7 GB/s on a single node, matching the network bandwidth of disaggregated storage, with a latency of less than 50 μs. Across 10 nodes StreamDedup shows an almost linear increase in throughput with less than 60 μs of latency.

Documents

Publisher URL: https://dl.acm.org/doi/10.1145/3799896download article:

BibTeX

@article{li2026streamdedup,
  author={Jiayong Li and Jonas Dann and Zhenhao He and Gustavo Alonso and Sai Rahul Chalamalasetti and Dejan Milojicic and Lance Evans and Alex Veprinsky and Runbin Shi},
  title={{StreamDedup: Distributed In-line Deduplication for Disaggregated Storage}},
  journal={ACM Trans. Reconfigurable Technol. Syst.},
  institution={ETH Zurich},
  year={2026},
  month={Mar.},
  location={New York, NY, USA},
  publisher={ACM},
  issn={1936-7406},
  note={Just Accepted},
  source={http://www.unixer.de/~htor/publications/},
}


serving: 216.73.216.78:39598	© Torsten Hoefler