Rich I/O-centric Derived Quantity Operators for Advanced In Situ Processing

Abstract

Scientific simulations produce increasingly large datasets that surpass practical limits for frequent input and output operations. Although performing analysis directly as data is produced (in situ) can mitigate this problem, embedding such analysis routines into complex scientific codes is often impractical or inflexible. We address this by introducing derived quantity operators that provide a modular and pluggable framework within high-level I/O libraries, enabling sophisticated in situ analysis to be performed transparently during data output with minimal modifications to the host application. The operators bind to persistent hierarchical storage through operator-declared metadata, allowing cross-timestep and cross-run analysis to be carried out seamlessly within the I/O path. We demonstrate the capabilities of our operators by integrating correctness validation and important scientific metrics, traditionally executed after simulation runs, directly into the I/O pathway as derived quantities. Our implementation illustrates how complex, non-mathematical analyses such as runtime validation can be efficiently performed during simulation, enabling early detection of errors. Our work broadens the range of in situ analyses achievable through derived quantity operators, promoting more flexible, intelligent, and reliable scientific workflows.

Tags