IRIS bridges parallel file systems and object stores through two bidirectional abstractions. Virtual Files expose object-store data through file semantics. Virtual Objects expose file-system data through key-value semantics. The ICS'18 paper reports more than 7x I/O performance improvement over the systems evaluated.
The Problem
Parallel file systems provide POSIX paths, offsets, and strong consistency. Object stores provide flat key-value namespaces and different consistency models. Workflows that span simulation and analytics therefore need conversion code or accept an interface poorly matched to part of the pipeline. IRIS performs this translation in middleware.
System Architecture
IRIS intercepts I/O calls and redirects them to the appropriate backend. Its two abstractions preserve the interface expected by the application:
Virtual Files present object-store data as paths, offsets, and byte streams. Virtual Objects present file-system data as keys and buckets. Applications can use their native interfaces across either backend.
Bidirectional mapping algorithms translate layout, chunking, and names between the two models. Tunable consistency preserves POSIX compatibility while allowing weaker guarantees where an application permits them. A background compactor periodically defragments files and compacts buffers and key-value sets left fragmented by updates.
Published Results
Bidirectional Storage Translation
Enosis maps files to objects and reports 2x to 27x gains over one-to-one translation in the DataCloud'17 paper. Syndesis maps objects to files and reports 3x improvement in the MTAGS'17 paper. IRIS combines both directions in a single middleware. Its ICS'18 evaluation reports more than 7x I/O performance improvement over the solutions evaluated.
Workflow Data Movement
Niobe coordinates workflow data through intermediate storage resources. The BigData'19 paper reports up to 10x acceleration over a no-integration baseline and up to 133% improvement over other integration solutions. HFlow models transfers as dynamically provisioned ByteFlows, with a 3x performance increase reported at CLUSTER'21.
Programmable and AI-Oriented I/O
LABIOS applies labels to asynchronous placement across heterogeneous storage. Its HPDC'19 paper reports up to 17x I/O performance and received the Karsten Schwan Best Paper Award. HCL supplies RDMA-based distributed containers, with 2x to 12x higher performance than BCL reported at CLUSTER'20. DLIO characterizes scientific deep-learning I/O; the CCGrid'21 Best Paper reports optimizations that reduce training time by up to 6.7x. Stimulus provides direct storage paths for TensorFlow and PyTorch, with a 5.3x speedup for Cosmic Tagger on Summit reported at CCGrid'22. LabStor composes customized userspace I/O stacks, with up to 60% improvement reported at SC'22. LuxIO provisions and configures storage services from workload characteristics.
Research software archive
Research Lineage
IRIS's bidirectional translation model informed Hermes, LABIOS, and IOWarp. The project also produced U.S. Patent 11,630,834 for label-based data representation I/O.
The IRIS source code (opens in a new tab) preserves the published middleware.