Coeus

Coeus computes derived quantities while scientific data is being produced. Its ADIOS2 plugin executes operators in transit, places raw and derived data across a storage hierarchy, and records enriched metadata for selective queries. Applications do not need to change their I/O code.

Research Question​

Scientific analysis often re-reads raw data to recompute the same quantities. Coeus studies when those quantities should be computed, what metadata they should retain, and where raw and derived data should reside. The goal is to reduce repeated computation and data movement without storing every possible result.

Published Results​

Hades: Context-Aware Active Storage​

The CCGrid'24 paper introduced the active-storage architecture:

• HDCalc compiles derived-quantity expressions into operation graphs.
• Hierarchical buffering scores data from dependencies, access history, prefetch predictions, and application priorities.
• Context-aware prefetching uses ADIOS2 step patterns to anticipate analysis reads.
• Enriched queries retrieve ranges, bounding boxes, statistics, and derived quantities through SQLite metadata.

On tested Gray-Scott producer-consumer workflows, the paper reports 3 to 4 times faster end-to-end analysis and a 20% gain from derived quantities alone. The system weak-scaled to 256 processes with 1 TB of total I/O.

Choosing What to Derive​

The strategy study compared three choices:

• Store computes and persists a value during the write path.
• Expression stores the formula and recomputes the value during analysis.
• Stats computes block-level statistics for selective retrieval.

For S3D, Stats reduced total time by up to 15% against Expression and used 25% less storage than Store. For E3SM with GPU computation, Stats ran 25% faster than the other strategies. The best choice depended on compute capacity, operator complexity, query selectivity, reader count, and network bandwidth.

State-Dependent Operators​

An eScience 2026 paper extends Coeus from independent operators, such as velocity magnitude and Q-criterion, to operators that retain state across timesteps. It compares offline, hybrid in-transit, and persistent online execution on Ares, Sandia DOoM, and ORNL Frontier.

The paper reports 40% lower I/O than the offline baseline, metadata overhead equal to 0.10% of execution time, about 26 seconds less validation overhead for a 166 GB WRF case with more than 190 variables, about 30% lower I/O time than the hybrid mode for Q-criterion on Xcompact3D, and strong scaling to 256 processes. See the conference listing (opens in a new tab).

Read the paper.

System Design​

Operational metadata describes data shapes, distributions, and locations. Coeus enriches that record with statistics, tags, operator state, and pointers to derived quantities. Four services add those capabilities to the ADIOS2 data path:

1. A derived-quantity language compiles equations into operation graphs.
2. A storage manager places data across memory and persistent tiers.
3. A metadata engine records location, statistics, tags, and operator state.
4. A query API retrieves raw and derived data through ranges and spatial bounds.

Coeus adds derived-quantity execution, tiered placement, enriched metadata, and prefetching to the ADIOS2 data path.

The published evaluations used Gray-Scott, S3D, E3SM, WRF, and Xcompact3D on systems that included Ares, Sandia DOoM, Frontier, and Perlmutter.

Continuing Research​

The August 2026 final report evaluates two emerging placement systems. Eris replays Recorder and DFTracer traces through an emulated hierarchy to train placement policies offline. Across six workload traces on one workstation, its training cost was 4 to 91 times lower. Aneris scores data importance during execution and reduced I/O time by 10% to 34% against heuristic tiered-placement policies on four unseen workloads.

Coeus's storage substrate and selected operator work continue in IOWarp. The Coeus adapter (opens in a new tab) remains public.