Before the
Explanation

We built Ahti because we needed a shared place for the records our systems produce. An agent might call a tool, a deployment might change, or a controller might react to something that happened earlier. We wanted to keep those records together without making every application invent its own way to identify, validate, and reference them.

We also wanted to be able to revisit an explanation. If an agent says why a deployment failed, another reader should be able to check the records it used. Saving the summary alone would leave us depending on the agent’s interpretation.

Ahti is the database we’ve been building for that work. We’re preparing to open-source it as shared infrastructure, rather than offer it as another commercial product.

What we changed

The early version included observability logic. As we wrote in our first Ahti post, we eventually removed it. We wanted different applications to use the same records without having to agree with an interpretation built into the database.

Ahti now stores claims from identified producers, checked against registered schemas. A correction is another record, so readers can still see the earlier account. References distinguish the inputs used to produce a result from the evidence cited to support it. Those can overlap, but using a log to generate a summary does not mean the log supports everything the summary says.

Ahti checks that these records have the right structure and come from an authorized producer. Deciding whether the claims are correct belongs to the people and applications reading them.

How we built it

The implementation is in Rust. Producers send records through a gRPC API, where Ahti checks their identity, namespace, and schema. Accepted records go through a write-ahead log. With the default settings, that log is synchronized before a new write is acknowledged.

We use Arrow to hold recent data in a columnar form in memory, then move older data into compressed Parquet files. DataFusion queries both with SQL. This gives us established formats and a query engine we don’t have to write ourselves.

What we do have to manage is the handoff between memory, files, and recovery. Ahti writes the Parquet files, publishes them in a manifest, and only then removes the corresponding hot data. After a restart, it uses the committed storage positions to avoid replaying records already covered by Parquet. Getting that ordering wrong can lose records or return them twice.

There is one writer process per data directory. That keeps append ordering local, but it also means Ahti currently has no replication or automatic failover.

What testing turned up

One recovery bug allowed a redacted payload to become visible again. The record was still there, so checking the row count would not have caught the problem. Our recovery checks compare identity, payload, position, and redaction state.

The fix tightened how checkpoints are validated and selected. Recovery checks their contents against committed metadata and refuses a damaged active checkpoint instead of silently choosing an older one that might have outdated redaction information.

We also handle the case where a write succeeds but its reply never reaches the producer. Retrying with the same record ID returns the stored identity without adding a second copy. The producer still has to check whether its retry matches the original content.

Alongside these tests, we use process-crash experiments and formal models of selected persistence and recovery rules. We check concrete execution traces against those models, but that does not prove the whole database correct. In particular, killing a process does not remove the operating system’s cache, so those tests don’t establish power-loss safety.

Where it’s going

Ahti is still in development. We want the upcoming open-source release to let other people inspect the implementation and use it independently of our products.

The aim is fairly practical: keep enough of the original record, within its retention and redaction rules, that someone can come back with a different question. The model or application that first read it shouldn’t determine what every later reader is able to find out.