Hermes
Hermes presents DRAM, NVMe, burst buffers, and parallel file systems as one distributed I/O buffering hierarchy. Its adapters intercept application I/O, while a placement engine selects tiers according to device performance and available capacity. The HPDC'18 paper reports more than 2x improvement over the buffering platforms evaluated. Hermes established the tiered data-management foundation later extended by IOWarp.
The Problem
Fast storage tiers can absorb bursts of application I/O, but each tier exposes different performance and capacity. Applications otherwise have to decide what to place on each device and when to move it. Hermes moves those decisions into middleware and coordinates the hierarchy as a shared buffering platform.
Research Contributions
Multi-Tiered Buffering Architecture (HPDC'18)
The HPDC'18 paper describes four data placement policies:
- Maximum Application Bandwidth (MaxBW), the default, places data in the highest possible tier, balancing bandwidth, latency, and capacity, and moves data down asynchronously to free space.
- Maximum Data Locality spreads data across all tiers at the same time according to a data dispersion unit, such as HDF5 chunks or files.
- Hot-data keeps frequently accessed data in the fast tiers as a cache.
- User-defined applies a buffering schema that the user supplies.
Hermes combines distributed metadata tracking with communication over RDMA-capable networks, and also runs on systems without RDMA.
Hierarchical Data Prefetching: HFetch (IPDPS'20)
HFetch pushes data through the hierarchy from the server side using file-access frequency and block recency. The IPDPS'20 paper reports 10 to 35% higher performance than the prefetchers evaluated and more than 50% over no prefetching. The JCST'20 journal article describes its integration with Hermes.
Hierarchical Compression: HCompress (IPDPS'20)
HCompress selects compression methods for each tier, balancing capacity savings against CPU cost. The IPDPS'20 paper reports 7x improvement over the tiered storage solutions evaluated.
Adaptive Data Replication: HReplica (BigData'20)
HReplica selects replication, compression, and storage tiers together. The BigData'20 paper reports 5.2x higher application performance than the replication schemes evaluated.
Multi-Tiered Data Eviction (SC'19)
Multi-tiered eviction policies select both the data to evict and its destination when a tier reaches capacity.
Architecture
Hermes organizes each machine around three resource classes:
- Node-local resources: DRAM and local NVMe or SSD devices.
- Shared resources: burst buffers available across compute nodes.
- Remote storage: disk-based parallel file systems such as Lustre and GPFS.
Vertical paths move data among local tiers. Horizontal paths coordinate data across nodes through shared storage. Each node runs a Hermes core with RDMA-capable communication and a multithreaded node manager.
Component Design
Hermes is a C++ middleware library that applications load through recompilation or LD_PRELOAD. Adapters support POSIX, HDF5, and MPI-IO. Optional hints provide application knowledge to the placement engine.
Performance Results
Library Evaluation
The HPDC'18 evaluation measures raw I/O throughput, metadata scalability, and inter-node communication.
Workload Evaluation
The evaluation also covers alternating compute-I/O, repetitive reads, VPIC, and HACC, with buffering interposed without application code changes.
Research Lineage and Software
Hermes's buffering architecture supports subsequent work in IOWarp, Coeus, and ChronoLog.
Hermes v1.2.1 (opens in a new tab) preserves the published system as a public release.