ZyronLake

A table format built for shared storage

Immutable .zyr data files against an append-only transaction log, one numbered version file per commit. Branches, time travel, versioned indexes, and clustering live in the format, on local disk, a shared filesystem, or object storage.

The format

An append-only log over immutable files

A ZyronLake table is a set of immutable .zyr data files, the same columnar format the database engine scans, recorded in an append-only transaction log. Two writers racing to commit both try to create the same numbered version file, and the filesystem decides the winner.

One numbered version file per commit. A checkpoint folds the log into a manifest, and data files are never rewritten.

One version file per commit

Every commit appends exactly one numbered version file to the transaction log. The table's history is the ordered list of those files.

Optimistic concurrency

Committing means creating the next numbered file with File::create_new. The call succeeds for exactly one writer, and that is the whole concurrency primitive.

Manifest checkpoints

Periodic checkpoints collapse the log into a manifest, so readers open one file instead of replaying the whole history.

Immutable data files

Data lives in immutable .zyr files, written once and referenced by versions. A commit changes which files are live, never their contents.

Versioning

Branches and time travel

Every commit produces a version, and every committed version stays addressable. Fork a branch, experiment in isolation, merge it back or drop it, and read the table as it was at any point in its history.

A branch forks copy-on-write from a committed version and merges back. Every committed version stays queryable.

Copy-on-write branches

A branch forks a table without rewriting its data files, and merge brings it back with conflict resolution.

Time travel

Query any committed version, by timestamp or by version number.

AS OF TIMESTAMPVERSION AS OF

Change feed

Ask for the changes between any two committed versions and read them as a feed.

Diff and patch

Compare two versions with diff, apply the difference with patch.

-- Branch the database, experiment in isolation, then merge or drop
CREATE BRANCH experiment FROM main;
USE BRANCH experiment;

-- Read a table as it was at a point in time
SELECT * FROM orders AS OF TIMESTAMP '2026-05-06 09:00:00';

Branching and time travel are plain SQL, over the same wire protocol as everything else.

Pruning

Skip files instead of reading them

Scan speed on shared storage comes from the files a query never opens. Three pruning structures sit on file metadata, and two clustering orders keep related rows together so the pruning has something to work with.

Versioned secondary indexes

Indexes are lake artifacts versioned alongside the data, so a snapshot can never pair with a stale index.

Bloom filters and zone maps

Per-file metadata rejects files whose contents cannot match a predicate, before any data is read.

Struct-of-arrays prune index

A struct-of-arrays index over file metadata drives skip decisions, with 94-100% file skip rates in practice.

Z-order and Hilbert clustering

Space-filling-curve clustering keeps related rows in the same files, with background maintenance keeping the layout current.

Operations

Constraints, maintenance, and where it runs

Constraints in the format

Unique and foreign-key constraints are enforced by the lake itself, with a clustered fast path. Enforcement lives in the format, not in a layer above it.

Maintenance operations

Table upkeep runs as first-class operations in the same SQL surface.

OPTIMIZEVACUUMFOLLOWREPAIR

Local disk

Shared filesystem

Object storage

The same format on every backing, versioned per commit and manifest-checkpointed.

Lake tables are first-class in the same SQL as heap tables, bridged by the same HybridScan operator that powers the HTAP database, and lake-holding nodes across the mesh share the same object-store backing, so one query reaches all of it.

Numbers

The numbers, trade-offs included

The lake is built for scans and pruning. Where the row heap is faster, that is published too, so the picture stays honest.

~154M rows/sec

Lake scan throughput

~199 µs

Point probe through an index

~100%

Zone-map row rejection

Where the row heap wins

Bulk load to queryable

~10.8x faster on the heap

Trickle load to queryable

~32.1x faster on the heap

Point lookup with a heap B+ tree index

~2.0x faster on the heap

On the other side of the same suite, a lake point lookup reads ~89x less I/O than the heap. Full tables and charts live on the performance page.

Running in the next five minutes

Prebuilt binaries for Linux and Windows, no external dependencies, one command to a running server.

Download Zyron View on GitHub