ChronoLog 1.0.0 released as a public beta
GRC released ChronoLog 1.0.0 (opens in a new tab) on June 6, 2024. It is the first public beta of ChronoLog, a distributed log store that GRC develops with the University of Chicago under an NSF CSSI framework grant. The release includes the three server components, a C++ client library with Python bindings, and example workloads. The team is asking users for feedback.
Why a distributed log store
Many systems now produce activity data continuously: scientific instruments and simulations, sensors on the Internet of Things, financial transactions, HPC job and resource monitors, and the telemetry and traces of distributed software. This data arrives as streams of small, time-stamped events from many sources at once, and it is most useful when it can be read back in order and queried by time range.
File systems and databases are not built for that pattern at scale. A shared log has to accept writes from many nodes, keep a single order across all of them and still serve readers while it grows. Ordering is the hard part. Most distributed logs order events through a central sequencer or other synchronization, which limits how fast they can grow.
ChronoLog takes a different approach. It orders events by physical time, using the synchronized clocks of the nodes, so that writers do not need to coordinate with each other. Order is guaranteed within the error bounds of the clock synchronization protocol. Log data is distributed horizontally across nodes and vertically across storage tiers, such as persistent memory and flash. Event ingestion is decoupled from persistence, so the servers pull data to lower tiers at their own pace while new events keep arriving.
What is in 1.0.0
The beta ships the ChronoLog server as three components:
- ChronoVisor orchestrates the deployment. It is the portal for clients, keeps the registry of recording processes and spreads the recording load across them.
- ChronoKeeper ingests log events from client processes at high speed, groups them into partial chunks of a log and passes them to the next tier.
- ChronoGrapher merges and sequences the partial chunks from many ChronoKeepers into complete, time-bounded chunks and archives them in persistent storage.
Together they form a working, distributed three-tier log recording system with configurable time-based data management. Events are totally ordered across client nodes within the clock synchronization bounds, and each client's identifiers and causal order are preserved. The system is elastic: ChronoKeeper and ChronoGrapher processes can join or leave a recording group while it runs.
On the client side, the release provides a multi-threaded C++ client library for highly concurrent event ingestion, Python bindings for that library, a command-line admin tool, and example client applications that simulate different logging workloads.
Status
ChronoLog 1.0.0 is marked as a pre-release on GitHub. It is meant for users who want to try the system on their own workloads and report problems, not yet for production use. Documentation and deployment instructions are on the project wiki and at chronolog.dev.
