Overview
Sluice is a Kafka-compatible streaming log with no broker disks. Agents speak the Kafka protocol and hold no durable state, a single sequencer orders everything through a metadata log, and an S3-compatible bucket (Cloudflare R2, MinIO or AWS S3) is the only durable layer.
Unmodified Kafka clients produce, consume and run consumer groups against it, and a deterministic simulator plus a history checker back the claim that no acknowledged record is ever lost.
Architecture
- Stateless agents. Agents speak the Kafka protocol to clients and keep no durable state of their own.
- Fencing without consensus. One sequencer orders writes through a metadata log fenced by object-store conditional writes, so only one writer can create each log entry.
- Raw batches. Client bytes are stored untouched; fetches patch the base offset, which Kafka's batch CRC does not cover.
- Deterministic simulation. The agent and sequencer are sans-I/O state machines that a simulator runs single-threaded in virtual time, injecting crashes, partitions and storage errors.
The full design is in the repository's design document.
Key design decisions
Each of these has a decision record in the repository:
- Metadata lives in object storage, not Postgres or etcd.
- One active sequencer.
- Raw Kafka batches are stored as-is and offsets are rewritten on read.
acks=1behaves likeacks=all.- Sans-I/O cores, so the whole system can run under deterministic simulation.
- Recovery starts from the newest snapshot found by LIST, and log continuity is confirmed after every append.
See the decision records for the reasoning and trade-offs.
Benchmarks
Measured on a laptop (Intel i5-12450H, 16 GB, Windows 11), so these are a reproducible baseline rather than production figures.
| Measure | Result |
|---|---|
| Simulated client operations, 0 invariant violations | 6.5M across 12,000 seeds with crashes, pauses, partitions, storage errors and lost responses |
| Bugs found by the simulator | 8, three of which lost acknowledged data |
| Live chaos test (real processes killed, including the active sequencer) | 71,133 idempotent sends, all acknowledged, 0 violations |
| Produce latency (250 ms flush window, 30–70 ms injected storage latency) | p50 389 ms, p99 690 ms at 2 MB/s |
| Throughput (two agents, one laptop) | 73 MB/s |
Every number links to the script and raw output that produced it in the repository's benchmarks page.