NPC 2024 keynote: Hermes, Coeus and ChronoLog
Xian-He Sun gave the first keynote of the IFIP International Conference on Network and Parallel Computing (NPC 2024) in Haikou, China, on December 7, 2024. Titled "Hermes, Coeus, and ChronoLog: Some big data I/O systems for network, parallel and distributed computing", the talk walked through three GRC systems that each attack a different part of the data access bottleneck.
The data bottleneck in AI
Sun started from deep learning. Its success depends on processing vast amounts of data, and computing systems are struggling to supply data fast enough, which turns I/O into the bottleneck. He argued that data access has become the main performance limit for network, parallel and distributed computing (NPC) when it serves AI, and that AI can in turn help relieve that limit. The talk looked at both directions, NPC for AI and AI for NPC, from the point of view of the I/O system.
Three systems
Hermes is a multi-tiered I/O buffering system. It optimizes how data moves through a deep memory and storage hierarchy, from memory through local flash and burst buffers to the parallel file system. The abstract describes it as serving both directions, NPC for AI and AI for NPC.
Coeus enriches the metadata that scientific data systems keep, and uses active storage, storage that can run computation, to calculate derived quantities and answer complex queries over scientific data. The goal is to let scientists ask questions of their data without first moving all of it to compute nodes. Coeus is a collaborative project funded by the DOE Office of Advanced Scientific Computing Research and led by Sandia National Laboratories, with GRC as a partner.
ChronoLog is a distributed, tiered log store for log data in cloud computing environments. It orders events by time across many nodes, so that logs from large distributed systems can be written and read at scale. ChronoLog is funded by an NSF CSSI framework grant, and its first public beta, version 1.0.0, was released in June 2024.
Sun closed with the open questions these systems raise and with ideas for computer system designs built for AI and big data. The talk extended the keynote he gave at IEEE/ACM CCGrid 2024 in May, which also covered Hermes, ChronoLog and Coeus.
Learn more
- NPC 2024 keynote page with the abstract (Internet Archive copy) (opens in a new tab)
- Hermes: A Heterogeneous-Aware Multi-Tiered Distributed I/O Buffering System (HPDC 2018)
- ChronoLog: A Distributed Shared Tiered Log Store with Time-based Data Ordering (MSST 2020)
- DOE funds Coeus
- ChronoLog 1.0.0 released as a public beta
- CCGrid 2024 keynote: AI and data in computer system research
- Project pages: Hermes, Coeus, ChronoLog
- Xian-He Sun's profile
