Spring 2026 papers: LLM inference meets the memory hierarchy
Between March and May 2026, GRC researchers and their collaborators presented four papers on one problem: long-context LLM inference is limited by how data moves through the memory hierarchy, not only by compute. The papers appeared at ASPLOS, EACL, DATE and IPDPS, and their answers range from a hardware prefetcher to KV cache pruning. A fifth paper, also at IPDPS, applies the same I/O lens to HPC workflows.
The I/O cost of attention (ASPLOS)
Self-attention, the core of transformer models, needs memory that grows with the square of the sequence length. At ASPLOS 2026 in Pittsburgh (March 22 to 26), Xiaoyang Lu and Xian-He Sun, with Boyu Long, Xiaoming Chen and Yinhe Han of the Institute of Computing Technology, Chinese Academy of Sciences, asked how far careful I/O analysis can take an accelerator for it.
The paper first derives an I/O analysis for tall-and-skinny matrix multiplication, which captures the dominant data movement of long-sequence attention. From that analysis the authors build AttenIO, an accelerator for exact long-sequence self-attention with three optimizations: analytically derived tiling and scheduling that minimize I/O, three-level overlap of communication and computation to hide I/O stalls, and parallel execution patterns for softmax. AttenIO runs 1.6 to 8.8 times faster than the best existing solutions the authors compared against. The authors argue that the method reaches beyond attention, as a principled way to guide I/O optimization in general.
Processing in memory for sparse attention (IPDPS)
LLM inference has two stages with different bottlenecks. The prefill stage, which processes the prompt, carries the quadratic cost of attention; the decoding stage, which produces one token at a time, is bound by memory. Sparse attention cuts the cost of prefill, but moving data between processor and memory still limits the GPU.
At IPDPS 2026 in New Orleans (May 25 to 29), Lu and Sun, with Lihan Hu and Peng Jiang of the University of Iowa, GRC's partner on UniMCC, and Hongrui Huang of Columbia University, presented PILOT. It places processing units inside memory to run hybrid sparse attention during prefill and both dense and sparse attention during decoding, and uses I/O analysis to choose tiling and scheduling inside the processing-in-memory hardware. Paired with a GPU, PILOT runs sparse prefill attention up to 19.96 times faster and decoding attention up to 57.11 times faster than the GPU baseline, and speeds up end-to-end long-sequence inference by up to 5.31 times compared with a conventional GPU system.
A prefetcher for memory-intensive workloads (DATE)
There is growing interest in running LLMs locally on CPUs, for privacy and low latency. Attention on a CPU moves large amounts of data between the cache hierarchy and DRAM, which is where hardware prefetchers help, but current prefetchers struggle with diverse and changing access patterns.
At DATE 2026 in Verona (April 20 to 22), Vadim Biryukov, Lu and Sun, with Zirui Liu of the University of Minnesota and Kaixiong Zhou of North Carolina State University, presented Zion. It combines independent temporal and spatial modules to cover a broad range of access patterns, and uses runtime feedback to keep prefetching efficient under memory pressure. In multi-core evaluations, Zion improved performance by up to 43.2% on SPEC benchmarks and 43.0% on self-attention workloads compared with the best existing prefetchers.
Pruning the KV cache one page at a time (EACL Findings)
The KV cache stores attention keys and values for earlier tokens so they are not recomputed, but it grows with every token. vLLM's PagedAttention keeps the cache in fixed-size blocks, and eviction methods that rank tokens by attention score do not fit, because the optimized attention kernels never return those scores.
PagedEviction, in the Findings of EACL 2026 (Rabat, March 24 to 29), ranks tokens using the key and value tensors instead. It evicts individual tokens during prefill and, during decoding, removes a whole block only after the newest block fills, so it needs no changes to the CUDA attention kernels. With Llama 3.2 1B and a 1,024-token cache budget, it reached up to 3,020 tokens per second, 37% more than keeping the full cache, and it cut latency by about 10% to 12% across 1B, 3B and 8B models. Jie Ye and Krishna Teja Chitty-Venkata of Argonne share first authorship, with Siddhisanket Raskar of PNNL, Sun, Anthony Kougkas, Murali Emani, Venkatram Vishwanath and Bogdan Nicolae.
The same lens on HPC workflows (IPDPS)
The second GRC paper at IPDPS turns I/O analysis to scientific workflows. Meng Tang, Kougkas and Sun, with Luanzheng Guo and Nathan Tallent of Pacific Northwest National Laboratory, argue that I/O-aware schedulers look at tasks one at a time and miss workflow-level behavior such as file reuse and producer-consumer data exchange. Their workflow-centric method correlates workflow structure with I/O metrics across six HPC workflows, covering 3,500 task instances, 1,387 unique files and 1.5 TB of I/O. The patterns they found point to better task placement, data movement and storage selection.