The Hidden Cost of Storage Mismatches: Why Faster I/O Per Task Can Slow Down Entire Workflows

Abstract

Data-intensive scientific workflows in high-performance computing exchange intermediate data between pipeline stages through shared storage such as parallel file systems, local SSDs, and burst buffers. Existing workflow schedulers select storage to maximize individual task throughput, implicitly assuming that faster I/O per task yields faster end-to-end execution. We show this assumption is wrong. We present an empirical study across two scientific workflows—genomics and climate tracking—on a production HPC cluster. By benchmarking storage systems across operation types, transfer sizes, parallelism levels, and data volumes, we uncover two non-obvious findings. First, data movement between storage tiers scales fundamentally differently from read and write operations: copy throughput plateaus or degrades at parallelism levels where reads and writes continue to scale, making cross-tier staging costs unpredictable from standard I/O benchmarks. Second, producer-consumer task pairs can exhibit performance inversion—configurations that maximize throughput for individual tasks introduce costly inter-stage data movement that increases total workflow time by up to 6.8x compared to storage-aware alternatives. Across both workflows, I/O accounts for 42–98% of total execution time, yet the penalty from storage mismatches between dependent tasks can exceed the compute cost of the tasks themselves. These patterns hold across fundamentally different access profiles, from large sequential transfers to metadata-heavy many-small-file workloads. Our results demonstrate that producer-consumer data dependencies must be treated as first-class constraints in workflow scheduling.

Tags

Links