This is the argument I want to make: the scarcest resource in modern research is no longer deep knowledge of a specific task. It is the ability to orchestrate compute, capital, data, and expert judgment to produce novel outcomes. The age of the specialist executor is giving way to the age of the agent orchestrator, and the researchers who internalize this shift earliest will define the next decade of scientific discovery.

Expertise Reimagined: From Knowing to Designing Intelligent Research Loops​

The traditional premium on comprehensive domain expertise, knowing every detail of a specific algorithm, every parameter of a simulation framework, every corner case of a storage protocol, is not disappearing. It is being repositioned. Deep knowledge remains the foundation, but its power is now amplified most when embedded into AI-driven workflows rather than held exclusively in a single researcher's head.

Consider the trajectory. A decade ago, the highest-impact researcher in an HPC lab was the person who could write the fastest MPI code, tune the most parameters, and run the most complex simulations end-to-end. That skill set still matters. But the researcher who will have the greatest impact over the next decade is the one who can design the research loop, the one who architects systems in which AI agents explore vast parameter spaces, analyze intermediate results, flag ambiguities for human review, and iterate toward solutions with minimal manual intervention.

This is not a hypothetical future. It is the design philosophy behind projects like IOWarp and DaYu, where our work has evolved from building tools that researchers operate to building platforms that researchers direct. The distinction is subtle but consequential: the former requires the researcher to be present at every step; the latter requires the researcher to be present at the right steps, the ones where human judgment, domain intuition, and creative hypothesis generation are genuinely irreplaceable.

The practical implication for PhD students is direct: invest in learning how to decompose research problems into components that can be delegated to intelligent systems, and components that require your irreplaceable expertise. The goal is not to automate yourself out of relevance, but to amplify your relevance by orders of magnitude.

Resource Orchestration as a Core Competency​

In a world where you can spin up thousands of AI agents or launch massively parallel simulations on demand, the bottleneck shifts from individual task completion to the strategic allocation of finite resources. Your HPC cluster has a fixed number of nodes. Your computational budget has a ceiling. Access to unique experimental datasets is constrained. And your own time, the hours you spend reviewing results, refining hypotheses, and making judgment calls, is the scarcest resource of all.

The modern researcher must become fluent in a new kind of resource management:

• Compute allocation: When do you burn GPU-hours on broad exploration versus targeted refinement? When is it worth running that 10,000-node simulation, and when is a well-designed surrogate model sufficient?
• Energy-aware scheduling: As energy costs become a first-class concern in HPC operations, the ability to schedule intensive workloads during off-peak windows or in regions with cheaper power is not just an operations problem, it is a research design problem.
• Human-in-the-loop budgeting: Perhaps the most underappreciated resource question is when to invest your own attention. An effective orchestrator knows which intermediate results to inspect personally and which to delegate to automated validation pipelines.

This is fundamentally an optimization problem, and it is one that HPC researchers are uniquely well-equipped to solve, if they recognize it as part of their job description. Resource orchestration is not administrative overhead. It is a core research competency, and it will increasingly separate high-impact labs from the rest.

At GRC, this thinking is reflected in how we approach projects like WisIO, which automates I/O bottleneck detection across entire workflow pipelines, and DTIO, which expresses I/O as composable, replayable tasks. Both embody the principle that the infrastructure itself should be intelligent enough to optimize resource utilization, freeing the researcher to focus on the science.

Agent Literacy: The New Foundational Skill​

Just as proficiency with version control, scripting, and statistical software became baseline expectations for researchers over the past two decades, literacy in designing, managing, and auditing AI agent workflows will become a foundational skill within the next five years. This goes well beyond prompt engineering. It is closer to product management for AI-driven research.

What does agent literacy look like in practice?

1. Decomposition: Breaking a complex research question into sub-tasks that AI can meaningfully assist with, and sub-tasks that require human expertise.
2. Objective specification: Defining clear, measurable objectives for AI agents, not just "analyze this dataset" but "identify statistically significant deviations from the baseline model at the 95% confidence level, and rank them by potential impact on the hypothesis."
3. Output auditing: Developing robust habits for evaluating AI-generated results. This means understanding failure modes, recognizing hallucination patterns, and building verification pipelines that catch errors before they propagate.
4. Iterative refinement: Treating AI workflows the way good engineers treat any system, with continuous testing, A/B experimentation, and iterative improvement. The first version of an AI-assisted research pipeline will not be the best version. The discipline of systematic refinement is what compounds impact over time.

This is why emerging work like the agentic memory direction of ChronoLog, which coordinates memory for multi-agent systems, and CLIO GenomIO, which develops context-aware AI agents for genomic data orchestration, represents a pedagogical contribution as well as a technical one. They are defining the design patterns that the next generation of researchers will need to internalize.

The Orchestrator's Mandate​

The argument I am making is not that deep technical skill is becoming less important. It is that deep technical skill, by itself, is no longer sufficient for maximum impact. The researchers who will lead the next era of scientific discovery will be the ones who combine three capabilities:

1. Domain expertise deep enough to know which questions matter and which results to trust.
2. Systems thinking broad enough to design intelligent pipelines that leverage heterogeneous resources, from HPC clusters to AI agents to human collaborators.
3. Orchestration discipline rigorous enough to manage the complexity that emerges when all of these capabilities are composed into a single research operation.

The capability of AI to undertake complex tasks is growing exponentially. The cost of compute continues to fall. The volume of data continues to rise. In this environment, the premium shifts decisively toward judgment, the ability to determine what to compute, when, with what resources, and to what end.

As you progress in your PhD, I encourage you to embrace this shift deliberately. Do not simply learn to use AI tools. Learn to direct them. Learn to build systems that make your research more efficient, more rigorous, and more ambitious than any individual, no matter how talented, could achieve alone.

This is the Age of the Agent Orchestrator. The infrastructure is being built. The question is whether you are ready to lead it.

Anthony Kougkas is the Co-Founder and Executive Director of the Gnosis Research Center at Illinois Institute of Technology, where he leads the center's systems-building efforts in intelligent storage, scientific workflows, and agentic data management. His research is supported by the National Science Foundation and the U.S. Department of Energy.

What MegaMmap Solved​

HPC applications, scientific simulations, and machine learning workloads routinely generate datasets that exceed available DRAM. The traditional solutions are buying more memory (expensive and unsustainable) or manually partitioning data across storage tiers (complex and error-prone). MegaMmap offered a third path: transparent memory virtualization that handles data placement automatically.

The system manages data across heterogeneous storage tiers, from DRAM to NVMe, SSD, and HDD, presenting applications with a unified memory abstraction. Three capabilities made this practical:

Transparent virtualization. Applications work with datasets as if they reside entirely in memory, using C++ vector-like interfaces. No explicit I/O management or partitioning logic required.

Workload-aware placement. Applications declare their access intent through transactions, and MegaMmap uses those hints to guide data placement across tiers, prefetching, and eviction.

Intent-based coherence. Instead of generic coherence protocols, MegaMmap provides workload-specific optimizations for read-only analytics, write-only simulations, and mixed access patterns.

SC'24 Results​

The results at SC'24 demonstrated that intelligent tiering can reduce memory requirements without sacrificing performance:

• 2.6x DRAM reduction with competitive performance on ML clustering workloads
• As much as 2x faster than Apache Spark on cosmological data analytics
• 45% less code compared to manual out-of-core implementations
• Competitive weak scaling to 768 processes

These numbers validated the central thesis: a well-designed memory virtualization layer can bridge the gap between available DRAM and dataset size without pushing complexity onto application developers.

Connecting to the GRC Portfolio​

MegaMmap reflects patterns established across multiple GRC projects:

Hermes taught us how to automatically manage heterogeneous storage hierarchies. MegaMmap extended those lessons from I/O buffering to full memory virtualization with distributed shared memory semantics. Learn more about Hermes.

LABIOS bridged HPC and Big Data storage paradigms. MegaMmap applies similar principles to bridge in-memory computing with storage-aware computing. Learn more about LABIOS.

ChronoLog provides time-ordered storage for activity and provenance data. MegaMmap addresses a different layer by expanding effective memory capacity across DRAM and storage. Learn more about ChronoLog.

Each project addresses a different facet of the data-centric computing challenge. Together, they form a coherent portfolio: intelligent systems that manage data movement across heterogeneous tiers so that applications and their developers don't have to.

What's Next​

As AI models grow larger and scientific datasets expand, the pressure on memory systems will intensify. MegaMmap's approach of transparent, workload-aware memory management positions it well for emerging challenges in LLM inference (where KV caches strain GPU memory), distributed training (where checkpointing I/O dominates), and scientific workflows (where multi-physics simulations exceed node-local DRAM).

• Source Code: github.com/grc-iit/mega_mmap (opens in a new tab)
• Original Paper: SC'24 proceedings (Logan, Sun, Kougkas)
• Questions? Reach out to grc@illinoistech.edu

At SC24, Luke Logan presented MegaMmap, a software-based distributed shared memory (DSM) technology that blurs the boundary between memory and storage. Developed by Luke Logan, Anthony Kougkas, and Xian-He Sun, MegaMmap lets data-intensive applications work with datasets larger than available DRAM through memory tiering, prefetching, and coherence optimization.

Luke Logan presenting MegaMmap research at SC24

The Memory Crisis in Modern HPC​

Scientific workloads such as cosmological simulation and machine learning increasingly process datasets larger than the DRAM of the nodes they run on. Out-of-core programming works around the limit, but it adds I/O code and complexity to every application.

The MegaMmap Solution​

MegaMmap is a software distributed shared memory (DSM) system that presents a unified, byte-addressable interface spanning multiple storage tiers, from DRAM to NVMe, SSD, and HDD.

Key Technical Contributions​

1. Infinite Memory Abstraction MegaMmap allows applications to present massive datasets as if they were in main memory, eliminating the need for explicit I/O management. A simple C++ vector-like interface enables developers to work with terabytes of data using familiar programming patterns.

2. Intelligent Tiering The system automatically manages data placement across heterogeneous storage based on access patterns, device performance characteristics, and application requirements. Hot data stays in fast DRAM, while cold data gracefully migrates to appropriate storage tiers.

3. Transactional Memory API Unlike traditional DSMs that must guess access patterns, MegaMmap allows applications to declare their intent through transactions. This enables:

• Optimized prefetching strategies
• Reduced coherence overhead
• Better data placement decisions

4. Intent-Aware Coherence MegaMmap provides workload-specific coherence optimizations for common HPC patterns:

• Read-only analytics (machine learning inference)
• Write-only simulations (scientific modeling)
• Mixed workloads (iterative algorithms)

Performance Results​

The SC'24 paper evaluates MegaMmap on HPC applications and AI workloads and reports:

Machine Learning and AI Workloads​

For KMeans clustering on large-scale cosmological datasets:

• 2.6x reduction in DRAM usage while maintaining competitive performance
• 45% less code compared to traditional out-of-core programming approaches
• As much as 2x faster than the Apache Spark implementations evaluated

Scientific Computing and Simulations​

Gray-Scott reaction-diffusion modeling and complex scientific simulations demonstrated:

• Datasets larger than DRAM: processed datasets exceeding available physical DRAM
• At least 20% faster performance than the tiered I/O systems evaluated through grid size L=2688
• Scaling across memory hierarchies from gigabytes to terabytes of data

Big Data Analytics and Data-Intensive Computing​

DBSCAN clustering on Gadget-4 cosmological simulation data performed competitively with the MPI-based implementation.

The Technology Behind MegaMmap: Architecture and Implementation Details​

MegaMmap's software-based distributed memory architecture consists of the following components, which provide transparent memory virtualization and intelligent data management across heterogeneous storage hierarchies:

Distributed Caching and Memory Hierarchy Management​

• Private Cache (pcache): Per-process DRAM cache providing ultra-low-latency access and reducing network overhead in distributed systems
• Shared Cache (scache): Distributed, tiered cache layer spanning all processes and storage tiers for efficient data sharing and locality optimization
• Asynchronous Operations: Intelligent overlap of computation with data movement to hide I/O latency and maximize hardware utilization

Intelligent Data Management and Placement Optimization​

• Prefetcher: Predictive prefetching engine that anticipates future memory accesses based on declared transaction patterns and access history
• Data Organizer: Sophisticated data placement algorithm that positions data across storage tiers based on performance scores, access frequency, and latency requirements
• Persistent Integration: Transparent, automatic staging of data to/from various storage backends (HDD, SSD, NVMe) and persistent formats

Advanced Coherence Protocols and Memory Consistency​

• Minimal Overhead: Optimized coherence mechanisms that avoid traditional distributed shared memory communication penalties
• Workload-Aware Optimization: Specialized coherence protocols tuned for HPC access patterns rather than generic cloud computing workloads
• Strong Consistency: Scheduling orders operations on the same page to provide read-after-write guarantees

Developer Experience: Simplified Programming Model for HPC​

MegaMmap provides a simplified programming interface and abstractions for memory management. Traditional out-of-core computing requires explicit data partitioning, complex I/O orchestration, and synchronization logic. MegaMmap eliminates this complexity through a clean, familiar C++ vector-like API that abstracts away storage hierarchy details. Consider this practical KMeans clustering example:

// Create a shared vector from a Parquet file
mm::Vector<Point3D> pts("/points.parquet");
pts.BoundMemory(MEGABYTES(1));  // Limit to 1MB DRAM
pts.Pgas(rank, nprocs);          // Partition across processes

// Begin read-only transaction
auto tx = pts.SeqTxBegin(pts.local_off(), pts.local_size(), MM_READ_ONLY);

float distance = 0;
for (Point3D p : tx) {
    distance += pow(NearestCentroid(p, ks), 2);
}

This simple interface eliminates the complexity of manual data partitioning, I/O management, and memory synchronization that typically plague out-of-core applications.

Performance Validation and Benchmarking​

Evaluation on a research cluster with hierarchical storage tiers (DRAM, NVMe, SSD, HDD):

Scalability Studies and Large-Scale Distributed Computing​

• Scaling to 768 processes across nodes
• Competitive performance parity with conventional MPI-based distributed implementations
• As much as 2x faster than Apache Spark

Memory Efficiency and DRAM Optimization​

• Data eviction and prefetching policies based on access patterns and memory pressure

Cost-Performance Analysis and Total Cost of Ownership​

• 1.8x performance improvement when optimizing with NVMe storage tiering

Open Source and Community: Contributing to HPC Software Infrastructure​

MegaMmap is now available as open-source software under an OSI-approved license, making its memory management techniques available to the HPC and research computing community.

The project includes documentation, reproducible benchmarks, and integration examples.

Get Started with MegaMmap: Installation, Documentation, and Support​

Researchers, HPC system administrators, and developers interested in exploring MegaMmap and implementing memory-virtualized computing systems can:

• Access the open-source MegaMmap GitHub repository (opens in a new tab) featuring complete source code, API documentation, and implementation guides for distributed memory systems.
• Read the complete SC24 research paper (opens in a new tab) for detailed evaluation results, benchmarking methodology, technical insights, and comparison with alternative distributed memory approaches.
• Contact the lead author, Dr. Luke Logan, for collaboration and technical questions.

Acknowledgments​

This research is supported by the National Science Foundation (NSF) (opens in a new tab) through grants CSSI-2104013 and Core-2313154, and by the U.S. Department of Energy (DOE) (opens in a new tab) Office of Science under Contract DE-SC0024593. The team also acknowledges the Chameleon Cloud (opens in a new tab) testbed for providing development infrastructure and resources that enabled this research in HPC and memory management systems.

The work​

The National Data Platform aims to make scientific data and computing resources easier for researchers to discover and use. IOWarp's CLIO architecture organizes data, metadata, and computational state as context that AI agents and scientific tools can act on, and the collaboration connects that context layer to NDP.

The first concrete result is an NDP server in CLIO Kit (opens in a new tab), IOWarp's collection of agent tools, released in June 2026. The two groups are now working to deploy CLIO and its ecosystem within the National Data Platform.

Learn more​

• IOWarp project page

The project​

The award abstract describes IOWarp as a modular data management platform for scientific workflows, particularly those that use AI. It targets data access times on hardware such as NVMe SSDs, CXL devices, and CPU-GPU co-designs, and builds on the team's prior NSF-funded research on multi-tiered storage, including Hermes.

Learn more​

• IOWarp project page

What it does​

DFTracer captures trace data across several layers of the software stack, from the application and its data loaders down to POSIX I/O, so that data loading stalls in training pipelines can be attributed to their source. The SC'24 paper reports lower runtime overhead and smaller traces than Score-P, Recorder, and Darshan on the workloads it studies, and tracing of dynamically spawned processes in AI workflows.

Learn more​

• Install pydftracer from PyPI (opens in a new tab)
• Read the DFTracer documentation (opens in a new tab)
• Read the paper

ChronoLog Project Overview​

ChronoLog is a scalable, high-performance distributed shared log store designed to handle the ever-growing volume, velocity, and variety of modern activity data. It is tailored for applications ranging from edge computing to high-performance computing (HPC) systems, offering a versatile solution for managing log data across diverse domains. Find out more at chronolog.dev (opens in a new tab).

Core Features​

• Multi-Tiered Storage: Uses multiple storage tiers (e.g., persistent memory, flash storage) to scale log capacity and optimize performance.
• High Concurrency: Supports multiple writers and multiple readers (MWMR) for efficient concurrent access to the log.
• Partial Data Retrieval: Enables efficient range queries for partial log processing, enhancing data exploration capabilities.
• Total Ordering: Orders log entries across distributed environments using synchronized physical clocks, within the error bounds of the node synchronization protocol.
• Synchronization-Free: Employs physical time for log ordering, avoiding expensive synchronization operations.
• Auto-Tiering: Automatically and transparently moves log data across storage tiers based on age and access patterns.
• Elastic I/O: Adapts to varying I/O workloads, ensuring efficient resource utilization and performance.

Applicability​

ChronoLog is designed to address the challenges of modern applications that generate and process vast amounts of log data. It is particularly well-suited for:

• Scientific Applications: Astrophysics, geoscience, materials science, cosmology, and more.
• Internet-of-Things (IoT): Efficiently stores, indexes, queries, and analyzes data from IoT devices.
• Financial Applications: Supports real-time monitoring, time-series analysis, and fraud protection.
• HPC Resource Management: Optimizes resource utilization in large-scale HPC systems.
• System Telemetry and Trace Recording: Captures and analyzes telemetry data from distributed systems.
• Application Performance Analysis: Facilitates performance monitoring and analysis for complex applications.

Key Technical Features​

• Physical Time-Based Ordering: An approach to achieving total ordering in distributed systems without synchronization overhead.
• 3D Log Distribution: A unique model that distributes log data horizontally across nodes and vertically across storage tiers.
• Decoupled Server-Pull Architecture: Separates event ingestion from event persistence for enhanced performance and scalability.

ChronoLog Release 1.0.0​

• ChronoLog Server
  • Implemented Components: ChronoVisor, ChronoKeeper, and ChronoGrapher
  • Multi-Tiered Distributed Storage: The fully functional distributed 3-tier log recording system is implemented to efficiently handle and manage log event streams. The system supports configurable time-based log data management.
  • Event Ordering: Total log event ordering across the client nodes is guaranteed within the bounds of error inherited from the node synchronization protocol. The client identifiers and the event causality order of the client application are retained. (Pipeline Data Model)
  • Elasticity: The system supports dynamic Recording Group membership for ChronoKeeper and ChronoGrapher processes
• ChronoLog Client API
  • Multi-threaded ChronoLog Client library implementation in C++ allows for highly concurrent log event ingestion and distributed recording
  • A set of example client applications and a command-line admin tool utilizing the C++ library to simulate various logging workloads
  • Python Bindings for C++ Client API

Development Team​

ChronoLog is being developed by a team of researchers and engineers at GRC and the University of Chicago.

Acknowledgements​

We gratefully acknowledge the support of the National Science Foundation (NSF) for funding this project. We also thank our collaborators from various scientific and engineering domains for their valuable insights and feedback.

In his keynote, Professor Sun addressed the complex challenges and emerging opportunities in computer system design brought about by the era of AI and big data. He highlighted the scalability issues caused by data access performance bottlenecks in current computer systems. This storage wall has led to a new performance analysis framework that prioritizes data throughput over floating-point computation.

Professor Sun emphasized that data systems are inherently more complex than computing systems. It was discussed the necessity of comprehensive redesigns and optimizations across operating systems, compilers, and hardware which must be designed to fully exploit emerging hardware technologies by making use of optimization methods such as data tiering, compression, indexing, and prefetching. Only through new architectures and performance models can future HPC systems mitigate the "storage wall" performance bottleneck in AI and big data computing.

He highlighted how these new challenges present fundamental research tasks, and opportunities for computer scientists and engineers. To emphasizes these opportunities, Professor Sun highlighted some of the research performed here at the Gnosis Research Center:

• Concurrent Average Memory Access Time (C-AMAT) Model (opens in a new tab): This model accurately reflects memory performance in modern high-performance computing systems, emphasizing the importance of addressing unhidden miss penalties in high concurrency storage operations.
• Layered Performance Matching (LPM) (opens in a new tab): A method for optimizing storage systems to achieve near-cache performance with large capacity. This method is implemented in the Hermes data input/output hierarchical management system, which enhances performance in scientific computing, big data applications, cloud computing, and deep learning.
• Hermes: Supported by the National Science Foundation, Hermes uses the C-AMAT model and LPM method, utilizing new storage technologies like NVRAM to enhance performance by 2-3 times. Hermes's success has led to collaboration projects such as ChronoLog, Coeus, and DTIO.

Professor Sun's team continues to optimize storage systems to enhance performance across various computing fields, reflecting the shift from computation-centric to data-centric computing. This ongoing work is essential in addressing the evolving needs of AI and big data applications, presenting substantial and ongoing challenges for computer scientists and engineers.

The work​

CARE is a concurrency-aware cache management framework published at HPCA 2023 by Xiaoyang Lu, Rujia Wang, and Xian-He Sun: CARE: A Concurrency-Aware Enhanced Lightweight Cache Management Framework. It starts from the observation that data access concurrency, in addition to locality, matters for cache performance, since a lower miss rate does not always mean better overall performance. CARE introduces the pure miss contribution (PMC) metric, which measures the cost of each outstanding miss while accounting for concurrency, and uses it to manage the cache.

Learn more​

• Dr. Xiaoyang Lu's profile
• Dr. Xian-He Sun's profile

From SCS Lab to GRC​

Dr. Xian-He Sun founded the Scalable Computing Software (SCS) Laboratory at Louisiana State University and brought it to Illinois Tech when he joined the Department of Computer Science in August 1999. His theoretical work includes the Sun-Ni Law for scalable parallel computing (1990) and, at Illinois Tech, the Concurrent Average Memory Access Time (C-AMAT) framework for memory system evaluation.

Over the next two decades, the lab's scope expanded from theoretical models into systems research. Hermes brought multi-tiered I/O buffering. LABIOS bridged the gap between HPC and Big Data storage paradigms, earning the Karsten Schwan Best Paper Award at HPDC 2019. ChronoLog introduced a distributed log store for scientific activity data. Each project reflected a consistent thesis: data movement, not computation, is the bottleneck of modern science.

By 2023, the operation had grown well beyond a single-PI lab. Research scientists, software engineers, and doctoral researchers were working across a portfolio of federally funded projects spanning NSF, DOE, LLNL, Argonne, and Pacific Northwest National Laboratory. The scope demanded a new structure.

Why "Gnosis"​

γνῶσις (gnōsis) is Greek for knowledge gained through investigation. The name captures what this center does: we produce knowledge by constructing and deploying the storage, I/O, and memory systems that scientific computing depends on.

What GRC Does​

GRC develops advanced solutions for high-performance computational capabilities at the intersection of scalable computing and data-driven discovery. Three objectives define the center's mission:

Advance scalable computing technologies. GRC builds middleware, runtime systems, and storage architectures for modern scientific workloads. Projects like IOWarp and Hermes directly address the I/O performance gap that limits computational science.

Develop tools for data-driven domains. Time-series analytics, workflow optimization, deep learning I/O pipelines. Projects like ChronoLog and DeepIO serve the growing convergence of HPC, AI, and Big Data.

Bridge researchers and infrastructure. GRC connects data scientists who need performance with systems architects who build it. The center's open-source releases, benchmarks, and national lab partnerships make that bridge tangible.

The Team​

Dr. Xian-He Sun serves as Founding Director, bringing the theoretical foundations and institutional relationships built since 1999 at Illinois Tech. Dr. Anthony Kougkas serves as Co-Founder and Executive Director, leading the systems-building effort that turned theory into funded software projects.

The research team includes PhD candidates, research scientists, and software engineers working across storage systems, memory architectures, deep learning I/O, and distributed computing. GRC also maintains active collaborations with Argonne National Laboratory, Lawrence Livermore National Laboratory, Pacific Northwest National Laboratory, and industry partners.

Open by Default​

Every major GRC system ships as open-source software. Hermes, ChronoLog, IOWarp, and LABIOS are all publicly available, and GRC researchers co-developed the DLIO benchmark with Argonne. This is deliberate: the center's mission requires that the tools we build reach the researchers who need them.

What's Ahead​

GRC's current portfolio spans intelligent I/O buffering, distributed time-series storage, deep learning data pipelines, memory system optimization, and large-scale workflow orchestration. As scientific computing enters an era where AI, HPC, and Big Data workloads converge on shared infrastructure, the need for principled data-centric systems research will only grow.

We built GRC to meet that need. Welcome.

Learn More​

• About GRC
• Our Team
• Active Projects

The project​

DTIO is a task-based I/O runtime. Its DataTask abstraction expresses data movement, ordering, and dependencies as composable tasks, and the runtime schedules format translation, staging, caching, and replay across HPC, Big Data, and ML data stacks.

Learn more​

• DTIO project page

Dr. Sun and ICPP​

ICPP is one of the longest-running conferences in parallel computing. Its 50th meeting took place in 2021, with Dr. Sun as a general chair (opens in a new tab). His ICPP record includes the Best Paper Award at ICPP 2001 in Valencia, Spain, with Kasidit Chanchio.

His research spans the memory-bounded speedup model known as Sun-Ni's Law (1990), the C-AMAT model of concurrent memory access published in IEEE Computer in 2014, and parallel I/O systems for hierarchical storage, the line of work the Gnosis Research Center continues.

Learn more​

• Professor Sun's profile

What the patent covers​

The patent describes a system and method for executing I/O tasks in a distributed computing system. The operation instructions of each I/O request are separated from its data and represented as a data label, which carries a unique identifier, the source or destination of the data, and the operation. Labels are pushed into a distributed label queue and dispatched to worker nodes according to a scheduling policy, and the workers execute the I/O tasks independently of the client applications.

Learn more​

• Patent page
• Google Patents entry (opens in a new tab)
• LABIOS project page

What is new​

The release notes list three changes:

• Tags. Users can associate related blobs under a tag and locate them through it. Hermes can also use tags to inform data placement for logically grouped data.
• Traits. Traits attached to a tag apply a set of operations to every blob in the group, for example compressing and encrypting it.
• Portability. Client programs no longer have to follow MPI design patterns, and installation issues on recent operating system versions are fixed.

Learn more​

• Hermes releases on GitHub (opens in a new tab)
• Hermes project page