HPC China 2019 keynote: high performance data access
Xian-He Sun gave a plenary keynote, "High Performance Data Access: the DMSH approach", at the National Annual Conference on High Performance Computing (HPC China 2019) in Hohhot, China, on August 22, 2019. The talk followed the memory wall down to storage and presented Hermes, GRC's system for buffering data across a deep memory and storage hierarchy (DMSH).
From the memory wall to the I/O wall
The talk opened with the trend lines behind Sun's earlier work on memory-bounded speedup (1990) and the memory wall (1994). Processor performance grew about 52% a year until 2004, and multi-core processors have pushed aggregate performance higher since. Memory performance grew about 9% a year and I/O about 6%. The gap between processors and storage keeps widening, so I/O has become the bottleneck of HPC. Parallel I/O systems, which stripe data across many file servers, help but are not enough.
A deep memory and storage hierarchy
HPC sites have begun installing new layers between memory and the parallel file system. One example in the talk was Cori, a Cray XC40 at NERSC that uses Cray's DataWarp burst buffer. New non-volatile memory devices and remote data access methods add further layers. Together they form a DMSH, a multi-level hierarchy in which a layer does not have to hold a copy of the data in the layer below.
The slides listed what was missing to use such a hierarchy well: software that manages heterogeneous tiers as a whole, intelligent placement of data among the tiers, automatic movement of data between them, native buffering support in HDF5, and the expertise that most users do not have. The stated goal was for the layers to be invisible to applications, with no loss of I/O performance.
Hermes
Sun's answer was to combine DMSH buffering with the HDF5 data format, which already carries rich information about the data it stores. Hermes, built with The HDF Group under an NSF grant, is a multi-tiered, distributed buffering platform. Its machine model has four layers: RAM, local NVMe or SSD, shared burst buffers and a remote parallel file system. Data moves vertically through the layers within a node and horizontally across nodes. Hermes components include a data placement engine that chooses a tier for each piece of data, a data organizer that carries out prefetching and eviction, a metadata manager, a cache manager with several replacement policies, and an application orchestrator that limits interference between applications sharing a tier.
The design drew on GRC papers from 2018: Hermes at ACM HPDC, IRIS at ACM ICS and the Harmonia I/O scheduler at IEEE Cluster, all by Anthony Kougkas, Hariharan Devarajan and Sun, with Jay Lofstead on Harmonia.
Learn more
- Keynote slides (PDF) (opens in a new tab)
- HPC China 2019 program (Internet Archive copy) (opens in a new tab)
- Hermes: A Heterogeneous-Aware Multi-Tiered Distributed I/O Buffering System (HPDC 2018)
- IRIS: I/O Redirection via Integrated Storage (ICS 2018)
- Harmonia: An Interference-Aware Dynamic I/O Scheduler (Cluster 2018)
- Hermes project page
- Xian-He Sun's profile
