Two journal articles on recoverable memory and workflow data flow
Two long-running lines of GRC research reached journals this summer. On July 8, ACM Transactions on Storage published NVREC, a recommendation system that trains in non-volatile memory and can recover after a failure. IEEE Transactions on Parallel and Distributed Systems accepted DataFlowDrs, a suite of tools from Pacific Northwest National Laboratory (PNNL) and GRC that finds and fixes data flow bottlenecks in HPC workflows. It will appear in the December issue.
NVREC: checkpoints that live in memory
Recommendation models hold very large embedding tables. Current training systems assume memory is volatile, so they periodically checkpoint model weights and embeddings to slower solid-state drives in case of failure. That design cannot take advantage of non-volatile memory (NVM), which keeps its contents when power is lost.
NVREC trains on a hybrid of DRAM and NVM and checkpoints in memory. It uses three techniques:
- It splits the embeddings so that only popular vectors sit in DRAM, which hides the higher latency of NVM.
- It keeps two versions of each vector in NVM, one persistent copy for crash recovery and one runtime copy for training updates, and alternates between them.
- It uses fine-grained parallelism to hide the cost of checkpointing the model parameters held in DRAM and on the GPU.
On synthetic and real-world datasets, NVREC trained 1.05 to 6.85 times faster than existing systems and checkpointed more than 5 times faster.
The authors are Ping Chen, Shuibing He, Peiyi Hong, Siling Yang, Tong Wu and Gang Chen of Zhejiang University, Xuechen Zhang of Kennesaw State University, and Xian-He Sun. He, now a professor at Zhejiang University, is a former postdoctoral researcher in Sun's group.
DataFlowDrs: diagnosing data flow in HPC workflows
Scientific workflows on HPC systems move large amounts of data between tasks, and the bottlenecks often appear in storage and I/O networks. Earlier work covered workflow coordination, scheduling and storage systems, but no tool suite could automatically find where data flow was holding a workflow back.
DataFlowDrs fills that gap with five parts:
- lightweight, high-resolution measurement and visualization of workflow data flow
- fast modeling that compresses repeated coordination patterns to shrink the analysis data
- interpretable models, generated automatically, that predict how data flow scales
- analysis that quantifies and ranks bottlenecks for different mixes of task parallelism and storage
- new schedules and resource assignments that resolve the bottlenecks
On a representative set of workflows, DataFlowDrs improved performance over baseline and earlier methods by choosing the right trade-off between task parallelism and storage performance. Student teams reproduced the work in the Student Cluster Competition at SC24, and the article summarizes their findings. The suite is open source and includes DaYu, the HDF5 data flow analysis tool that Meng Tang of GRC first-authored at IEEE Cluster 2024.
The authors are Nathan Tallent, Jesun Firoz, Zhen Peng and Luanzheng Guo of PNNL, together with Tang, Anthony Kougkas and Sun of GRC.
Update (August 2026): The partnership with PNNL continues in Portus, a DOE Genesis Mission project led by PNNL in which GRC is a co-PI. See GRC joins DOE's Genesis Mission with Argonne and PNNL.