Skip to main content

DiRecMR

Lifecycle
Completed
Period
2017–2018

DiRecMR examines how differences between map and reduce phases affect resource use, task execution, speculation, and failure handling. The NSF-supported research (opens in a new tab) combines theoretical analysis, simulation, and system design to address those differences in MapReduce systems.

The project formulates a Markov-chain model of Hadoop MapReduce container transitions and a fork-join model of map and reduce task queues. These models support analysis of resource management and task scheduling in large workloads.

The research aims to broaden the scope of task speculation and improve resilience while limiting failure amplification. Its target platforms include YARN and Spark, with the broader objective of improving efficiency and fault handling in distributed analytics environments.

Acknowledgment
National Science Foundation

This material is based upon work supported by the National Science Foundation under Grant No. CCF-1744317. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Explore all projects